Apache Avro

• Row-oriented (vs Parquet columnar) — better for streaming, individual row writes
• Schema: JSON-defined, embedded in every file / Kafka message
• Schema Registry: shared central schemas for Kafka (Confluent standard)
• Evolution: add optional fields backward-compatible
• Languages: Java, Python, C++, Go, Rust, JavaScript

{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "id", "type": "long"},
    {"name": "email", "type": "string"},
    {"name": "age", "type": ["null", "int"], "default": null}
  ]
}

# Python (fastavro)
import fastavro
with open('users.avro', 'wb') as out:
  fastavro.writer(out, schema, records)

• Parquet
• Data Lake

Apache Avro is a schema-first data serialization framework primarily used in big data applications. It enables the serialization of data in a compact binary format, ensuring efficient data exchange and storage. Avro schemas are defined using JSON, allowing for easy integration and evolution of data structures. This makes it particularly suitable for use with Apache Hadoop and Kafka, where data processing and streaming are critical.

Apache Avro employs a schema-first approach, meaning that data is serialized according to a predefined schema. This design choice offers several advantages:

• Data Integrity: By enforcing a schema, Avro ensures that the data conforms to a specific structure, minimizing errors during data processing.
• Backward and Forward Compatibility: Avro supports schema evolution, allowing for changes in the data structure without breaking compatibility with previous versions.
• Efficient Serialization: Avro's binary format is compact, leading to reduced storage requirements and faster serialization/deserialization processes.

To define an Avro schema, you use JSON syntax. Here’s a basic example:

{
  "type": "record",
  "name": "User",
  "fields": [
    { "name": "name", "type": "string" },
    { "name": "age", "type": "int" },
    { "name": "emails", "type": { "type": "array", "items": "string" } }
  ]
}

This schema defines a User record with three fields: name (string), age (integer), and emails (array of strings).

Apache Avro is widely used in various data-intensive applications, particularly in conjunction with big data tools like Apache Hadoop and Apache Kafka. Here are some practical use cases:

1. Data Serialization in Streaming Applications: In a Kafka-based architecture, Avro is used to serialize messages being sent between producers and consumers. This ensures that the data format remains consistent across services. For example, when producing messages to a Kafka topic, you can use the Avro serializer:

1. Data Storage in Hadoop: Avro files can be directly stored in HDFS (Hadoop Distributed File System), making it easy to work with large datasets. The command to convert a JSON file to Avro format using the Avro command-line tools is as follows:

1. Interoperability Between Languages: Avro supports multiple programming languages, including Java, Python, and C++. This allows for seamless data exchange across different systems. For instance, an Avro-encoded data file can be read in Python as follows:

These examples demonstrate how Avro's schema-first design enhances data handling in modern applications, ensuring efficiency and reliability in data processing workflows.