MotherDuck

If you’re new to Unstructured, read this note first.

Before you can create a destination connector, you must first sign in to your Unstructured account:

• If you do not already have an Unstructured account, go to https://unstructured.io/contact and fill out the online form to indicate your interest.
• If you already have an Unstructured account, go to https://platform.unstructured.io and sign in by using the email address, Google account, or GitHub account that is associated with your Unstructured account.

After you sign in, the Unstructured user interface (UI) appears, which you use to get your Unstructured API key. To learn how, watch this 40-second how-to video.

After you create the destination connector, add it along with a source connector to a workflow. Then run the worklow as a job. To learn how, try out the hands-on Workflow Endpoint quickstart, go directly to the quickstart notebook, or watch the two 4-minute video tutorials for the Unstructured Python SDK.

You can also create destination connectors with the Unstructured user interface (UI). Learn how.

If you need help, reach out to the community on Slack, or contact us directly.

You are now ready to start creating a destination connector! Keep reading to learn how.

Send processed data from Unstructured to MotherDuck.

The requirements are as follows.

• A MotherDuck account.

• A MotherDuck access token for the account.

• A database in the account.

  • Create a database.
  • List available databases.

  You can run commands to manage MotherDuck databases, schemas, tables, and more in the MotherDuck UI or for example by connecting to MotherDuck with the DuckDB CLI.

• A schema in the target database.

  • Create a schema.

  • You can list available schemas and their parent catalogs by running the following command in the MotherDuck UI or the DuckDB CLI:

    SELECT * FROM information_schema.schemata;
    

  The MotherDuck connector uses the default schema name of main if not otherwise specified.

• A table in the target schema.

  • Create a table.

  • You can list available tables in a schema by running the following commands in the MotherDuck UI or the DuckDB CLI, replacing the target catalog and schema names:

    USE <catalog_name>.<schema_name>;
    SHOW TABLES;
    

  The MotherDuck connector uses the default table name of elements if not otherwise specified.

  For maximum compatibility, Unstructured recommends the following table schema:

  CREATE TABLE elements (
      id VARCHAR,
      element_id VARCHAR,
      text TEXT,
      embeddings FLOAT[],
      type VARCHAR,
      system VARCHAR,
      layout_width DECIMAL,
      layout_height DECIMAL,
      points TEXT,
      url TEXT,
      version VARCHAR,
      date_created INTEGER,
      date_modified INTEGER,
      date_processed DOUBLE,
      permissions_data TEXT,
      record_locator TEXT,
      category_depth INTEGER,
      parent_id VARCHAR,
      attached_filename VARCHAR,
      filetype VARCHAR,
      last_modified TIMESTAMP,
      file_directory VARCHAR,
      filename VARCHAR,
      languages VARCHAR[],
      page_number VARCHAR,
      links TEXT,
      page_name VARCHAR,
      link_urls VARCHAR[],
      link_texts VARCHAR[],
      sent_from VARCHAR[],
      sent_to VARCHAR[],
      subject VARCHAR,
      section VARCHAR,
      header_footer_type VARCHAR,
      emphasized_text_contents VARCHAR[],
      emphasized_text_tags VARCHAR[],
      text_as_html TEXT,
      regex_metadata TEXT,
      detection_class_prob DECIMAL,
      partitioner_type VARCHAR
  );
  

  You can list the schema of a table by running the following commands in the MotherDuck UI or the DuckDB CLI, replacing the target catalog, schema, and table names:

  USE <catalog_name>.<schema_name>;
  DESCRIBE TABLE <table_name>;
  

To create a MotherDuck destination connector, see the following examples.

Python SDK

import os

from unstructured_client import UnstructuredClient
from unstructured_client.models.operations import CreateDestinationRequest
from unstructured_client.models.shared import (
    CreateDestinationConnector,
    DestinationConnectorType,
    MotherDuckDestinationConnectorConfigInput
)

with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
    response = client.destinations.create_destination(
        request=CreateDestinationRequest(
            create_destination_connector=CreateDestinationConnector(
                name="<name>",
                type=DestinationConnectorType.MOTHERDUCK,
                config=MotherDuckDestinationConnectorConfigInput(
                    database="<database>",
                    db_schema="<db-schema>",
                    table="<table>",
                    md_token="<md-token>"
                )
            )
        )
    )

    print(response.destination_connector_information)

Replace the preceding placeholders as follows:

• <name> (required): A unique name for this connector.
• <database> (required): The name of the target MotherDuck database.
• <db-schema> (required): The name of the target schema within the database.
• <table> The name of the target table within the schema. By default, this table is named elements if not otherwise specified.
• <md-token> (required): The access token value within the MotherDuck account that has the appropriate access to the target database, schema, and table.