使用模拟 Cloud Storage 客户端编写测试

Rust 版 Cloud 客户端库提供了一种对实际客户端实现进行桩处理的方法，以便您注入模拟进行测试。您可以使用模拟来编写受控、可靠的单元测试，这些测试不会进行网络调用，也不会产生费用。

在本文档中，您将了解以下主题：

  • 如何使用 Cloud Storage Storage 客户端编写可测试的接口。
  • 如何模拟读取操作。
  • 如何模拟写入操作。
  • Storage 客户端的设计与其他 Google Cloud客户端不同的原因。

依赖项

将 Cloud Storage 客户端库添加到 Cargo.toml 文件中：

cargo add google-cloud-storage

如需编译和运行本文档中的单元测试，请添加以下开发依赖项：

cargo add --dev anyhow bytes mockall
cargo add --dev tokio --features macros,rt-multi-thread

编写可测试的接口

如果您不对存储代码进行单元测试，则可以直接根据 Storage 编写接口，其默认桩类型参数是客户端的生产实现：

pub async fn my_function(_client: Storage) {}

如需使用模拟客户端测试代码，请根据受 gcs::stub::Storage + 'static 约束的通用类型参数 T 编写接口：

pub async fn my_testable_function<T>(_client: Storage<T>)
where
    T: gcs::stub::Storage + 'static,
{
}

模拟读取操作

如需模拟 read_object 请求，请完成以下步骤：

  1. 编写一个接受通用 &Storage<T> 客户端的应用函数。 例如，以下函数会下载一个对象并统计其中包含的换行符数量： 

    // Downloads an object from GCS and counts the total lines.
pub async fn count_newlines<T>(
    client: &Storage<T>,
    bucket_id: &str,
    object_id: &str,
) -> gcs::Result<usize>
where
    T: gcs::stub::Storage + 'static,
{
    let mut count = 0;
    let mut reader = client
        .read_object(format!("projects/_/buckets/{bucket_id}"), object_id)
        .set_generation(42)
        .send()
        .await?;
    while let Some(buffer) = reader.next().await.transpose()? {
        count += buffer.into_iter().filter(|c| *c == b'\n').count();
    }
    Ok(count)
}

  2. 创建一个辅助函数，通过将载荷传递给 ReadObjectResponse::from_source 来构造伪造的 ReadObjectResponse 字节流。此方法接受与 Storage::write_object 相同的载荷类型： 

    fn fake_response(size: usize) -> ReadObjectResponse {
    let mut contents = String::new();
    for i in 0..size {
        contents.push_str(&format!("{i}\n"))
    }
    ReadObjectResponse::from_source(ObjectHighlights::default(), bytes::Bytes::from(contents))
}

  3. 定义一个实现 gcs::stub::Storage 的模拟结构体。本文档使用 mockall crate，但您可以在测试中使用其他模拟框架： 

    mockall::mock! {
    #[derive(Debug)]
    Storage {}
    impl gcs::stub::Storage for Storage {
        async fn read_object(&self, _req: ReadObjectRequest, _options: RequestOptions) -> Result<ReadObjectResponse>;
        async fn write_object_buffered<P: StreamingSource + Send + Sync + 'static>(
            &self,
            _payload: P,
            _req: WriteObjectRequest,
            _options: RequestOptions,
        ) -> Result<Object>;
        async fn write_object_unbuffered<P: StreamingSource + Seek + Send + Sync + 'static>(
            &self,
            _payload: P,
            _req: WriteObjectRequest,
            _options: RequestOptions,
        ) -> Result<Object>;
    }
}

  4. 编写一个单元测试，用于设置对模拟对象的预期、使用 Storage::from_stub 构造客户端并调用 count_newlines： 

    #[tokio::test]
async fn test_count_lines() -> anyhow::Result<()> {
    let mut mock = MockStorage::new();
    mock.expect_read_object().return_once({
        move |r, _| {
            // Verify contents of the request
            assert_eq!(r.generation, 42);
            assert_eq!(r.bucket, "projects/_/buckets/my-bucket");
            assert_eq!(r.object, "my-object");

            // Return a `ReadObjectResponse`
            Ok(fake_response(100))
        }
    });
    let client = gcs::client::Storage::from_stub(mock);

    let count = count_newlines(&client, "my-bucket", "my-object").await?;
    assert_eq!(count, 100);

    Ok(())
}

模拟写入操作

如需模拟 write_object 请求，请完成以下步骤：

  1. 编写一个应用函数，该函数使用通用 &Storage<T> 客户端上传对象： 

    // Uploads an object to GCS.
pub async fn upload<T>(client: &Storage<T>, bucket_id: &str, object_id: &str) -> gcs::Result<Object>
where
    T: gcs::stub::Storage + 'static,
{
    client
        .write_object(
            format!("projects/_/buckets/{bucket_id}"),
            object_id,
            "payload",
        )
        .set_if_generation_match(42)
        .send_unbuffered()
        .await
}

  2. 使用上一部分中定义的 MockStorage 结构体，编写一个单元测试，用于在模拟对象上配置预期、使用 Storage::from_stub 构造客户端并调用 upload： 

    #[tokio::test]
async fn test_upload() -> anyhow::Result<()> {
    let mut mock = MockStorage::new();
    mock.expect_write_object_unbuffered()
        .return_once(
            |_payload: Payload<BytesSource>, r, _| {
                // Verify contents of the request
                assert_eq!(r.spec.if_generation_match, Some(42));
                let o = r.spec.resource.unwrap_or_default();
                assert_eq!(o.bucket, "projects/_/buckets/my-bucket");
                assert_eq!(o.name, "my-object");

                // Return the object
                Ok(Object::default()
                    .set_bucket("projects/_/buckets/my-bucket")
                    .set_name("my-object")
                    .set_generation(42))
            },
        );
    let client = gcs::client::Storage::from_stub(mock);

    let object = upload(&client, "my-bucket", "my-object").await?;
    assert_eq!(object.generation, 42);

    Ok(())
}

预期详情

由于 upload 函数会调用 send_unbuffered，因此请对相应的 expect_write_object_unbuffered 方法设置预期：

mock.expect_write_object_unbuffered()

mockall::mock! 宏会将每个泛型实例化视为一个不同的方法。在闭包签名中指定具体载荷类型 Payload<BytesSource>，以便编译器可以匹配方法预期：

|_payload: Payload<BytesSource>, r, _| {

设计理由

以下部分介绍了 Storage 客户端为何为其桩使用通用类型参数，而不是动态调度。

其他客户

大多数客户端（例如 StorageControl）在内部保留其桩 trait 的引用计数 (Arc) 且与 dyn 兼容的实现。它们使用动态分派将请求从客户端转发到底层桩，底层桩可以是生产实现，也可以是模拟。

由于这些客户端使用动态分派，因此编译器无需在编译时间解析具体桩类型，并且客户端结构体无需为其桩提供泛型类型形参。

Storage 客户端

在 Rust 中，与 dyn 兼容的 trait 不能具有泛型方法或返回未装箱的 impl Future 值：

  • write_object_buffered 和 write_object_unbuffered 方法是针对载荷来源类型的通用方法。
  • gcs::stub::Storage 特征方法会直接返回 impl Future，而不是对返回的 future 进行装箱。

如果对 Storage 客户端使用动态分派，则需要在每次调用时对载荷和 future 进行装箱，从而增加数据路径上的堆分配量和动态分派开销。为了避免这种开销，Storage 客户端改为使用类型形参来具体实现 gcs::stub::Storage 特征。

完整的应用代码和测试套件

以下代码将应用函数和单元测试组合成一个完整示例：

use gcs::client::Storage;
use gcs::model::Object;
use google_cloud_storage as gcs;

// Downloads an object from GCS and counts the total lines.
pub async fn count_newlines<T>(
    client: &Storage<T>,
    bucket_id: &str,
    object_id: &str,
) -> gcs::Result<usize>
where
    T: gcs::stub::Storage + 'static,
{
    let mut count = 0;
    let mut reader = client
        .read_object(format!("projects/_/buckets/{bucket_id}"), object_id)
        .set_generation(42)
        .send()
        .await?;
    while let Some(buffer) = reader.next().await.transpose()? {
        count += buffer.into_iter().filter(|c| *c == b'\n').count();
    }
    Ok(count)
}

// Uploads an object to GCS.
pub async fn upload<T>(client: &Storage<T>, bucket_id: &str, object_id: &str) -> gcs::Result<Object>
where
    T: gcs::stub::Storage + 'static,
{
    client
        .write_object(
            format!("projects/_/buckets/{bucket_id}"),
            object_id,
            "payload",
        )
        .set_if_generation_match(42)
        .send_unbuffered()
        .await
}

#[cfg(test)]
mod tests {
    use super::{count_newlines, upload};
    use gcs::Result;
    use gcs::model::{Object, ReadObjectRequest};
    use gcs::model_ext::{ObjectHighlights, WriteObjectRequest};
    use gcs::read_object::ReadObjectResponse;
    use gcs::request_options::RequestOptions;
    use gcs::streaming_source::{BytesSource, Payload, Seek, StreamingSource};
    use google_cloud_storage as gcs;

    mockall::mock! {
        #[derive(Debug)]
        Storage {}
        impl gcs::stub::Storage for Storage {
            async fn read_object(&self, _req: ReadObjectRequest, _options: RequestOptions) -> Result<ReadObjectResponse>;
            async fn write_object_buffered<P: StreamingSource + Send + Sync + 'static>(
                &self,
                _payload: P,
                _req: WriteObjectRequest,
                _options: RequestOptions,
            ) -> Result<Object>;
            async fn write_object_unbuffered<P: StreamingSource + Seek + Send + Sync + 'static>(
                &self,
                _payload: P,
                _req: WriteObjectRequest,
                _options: RequestOptions,
            ) -> Result<Object>;
        }
    }

    fn fake_response(size: usize) -> ReadObjectResponse {
        let mut contents = String::new();
        for i in 0..size {
            contents.push_str(&format!("{i}\n"))
        }
        ReadObjectResponse::from_source(ObjectHighlights::default(), bytes::Bytes::from(contents))
    }

    #[tokio::test]
    async fn test_count_lines() -> anyhow::Result<()> {
        let mut mock = MockStorage::new();
        mock.expect_read_object().return_once({
            move |r, _| {
                // Verify contents of the request
                assert_eq!(r.generation, 42);
                assert_eq!(r.bucket, "projects/_/buckets/my-bucket");
                assert_eq!(r.object, "my-object");

                // Return a `ReadObjectResponse`
                Ok(fake_response(100))
            }
        });
        let client = gcs::client::Storage::from_stub(mock);

        let count = count_newlines(&client, "my-bucket", "my-object").await?;
        assert_eq!(count, 100);

        Ok(())
    }

    #[tokio::test]
    async fn test_upload() -> anyhow::Result<()> {
        let mut mock = MockStorage::new();
        mock.expect_write_object_unbuffered()
            .return_once(
                |_payload: Payload<BytesSource>, r, _| {
                    // Verify contents of the request
                    assert_eq!(r.spec.if_generation_match, Some(42));
                    let o = r.spec.resource.unwrap_or_default();
                    assert_eq!(o.bucket, "projects/_/buckets/my-bucket");
                    assert_eq!(o.name, "my-object");

                    // Return the object
                    Ok(Object::default()
                        .set_bucket("projects/_/buckets/my-bucket")
                        .set_name("my-object")
                        .set_generation(42))
                },
            );
        let client = gcs::client::Storage::from_stub(mock);

        let object = upload(&client, "my-bucket", "my-object").await?;
        assert_eq!(object.generation, 42);

        Ok(())
    }
}

运行 cargo test 以执行 test_count_lines 和 test_upload：

cargo test