Introduction to datasets

BigQuery datasets let you organize and control access to your tables and views. You can also use datasets to specify storage locations, manage retention policies, configure cross-region replication and disaster recovery, set default table properties, and connect to external data sources.

Datasets

A dataset is contained within a specific project. Datasets are top-level containers that are used to organize and control access to your tables and views. A table or view must belong to a dataset, so you need to create at least one dataset before loading data into BigQuery. Use the format projectname.datasetname to fully qualify a dataset name when using GoogleSQL, or the format projectname:datasetname to fully qualify a dataset name when using the bq command-line tool.

Location

You specify a location for storing your BigQuery data when you create a dataset. For a list of BigQuery dataset locations, see BigQuery locations. BigQuery stores your data in the selected location. For example, if you choose EU or an EU-based region for the dataset location, your Core BigQuery Customer Data resides in the EU.

After you create the dataset, its location cannot be changed directly. However, you can use cross-region dataset replication to replicate the dataset to a different location and promote the secondary replica.

If you don't explicitly specify a location, the location is determined in one of the following ways:

  • The location of the datasets referenced in the request. For example, if a query references a table or view in a dataset stored in the asia-northeast1 region, the query job runs in asia-northeast1.
  • The region specified for a connection referenced in a request.
  • The location of a destination table.

If the location isn't explicitly specified, and it can't be determined from the resources in the request, the default location is used. If default location isn't set, the job runs in the US multi-region.

Cross-region dataset replication

With BigQuery cross-region dataset replication, you can set up automatic asynchronous replication of a dataset between two different regions or multi-regions. Dataset replication provides geo-redundancy, lets you run read queries against the secondary replica in another region, and lets you migrate datasets across regions. For more information, see Cross-region dataset replication.

Managed disaster recovery

For business continuity during total regional outages, you can configure BigQuery managed disaster recovery. Managed disaster recovery pairs cross-region dataset replication with Enterprise Plus edition failover reservations to manage both storage and compute failover. It offers both hard failover for regional outages and soft failover for synchronized maintenance. For more information, see Managed disaster recovery and Migrate to managed disaster recovery.

Data retention

Datasets use time travel in conjunction with the fail-safe period to retain deleted and modified data. By default, the time travel window is 7 days, but you can configure it from 2 to 7 days. Time travel lets you query historical data at a point in time or restore deleted or expired tables and datasets. After the time travel window ends, data enters an unconfigurable 7-day fail-safe period for emergency recovery by Cloud Customer Care. For more information, see Data retention with time travel and fail-safe.

External datasets

In addition to BigQuery datasets, you can create external datasets, which are links to external data sources:

External datasets are also known as federated datasets; both terms are used interchangeably.

Once created, external datasets contain tables from a referenced external data source. Data from these tables aren't copied into BigQuery, but queried every time they are used. For more information, see Spanner federated queries.

Limitations

BigQuery datasets are subject to the following limitations:

  • The dataset location can only be set at creation time. After a dataset is created, its location cannot be changed.
  • All tables that are referenced in a query must be stored in datasets in the same location.

  • External datasets don't support table expiration, replicas, time travel, default collation, default rounding mode, or the option to enable or disable case-insensitive table names.

  • When you copy a table, the datasets that contain the source table and destination table must reside in the same location.

  • Dataset names must be unique for each project.

Quotas

For more information on dataset quotas and limits, see Quotas and limits.

Security

To control access to datasets in BigQuery, see Controlling access to datasets. For information about data encryption, see Encryption at rest.

What's next