Rafael Caricio

An Exercise in Agentic Coding: AV1 Encoder from Scratch in Rust

2026-02-16T00:00:00+00:00

It's a contentious time to be a software engineer. Some of us really love agentic coding, others hate it with passion. Some say it makes them super-productive, for others it's just useless slop. Little over a year ago, I thought agentic coding was just a fad, I was even annoyed by the AI auto-complete. That is, until I tried Cline (VS Code plugin)</a> in November 2024. It was then that I realized tools for writing software were changing and I needed to start paying closer attention. I'm an extremely curious person and anyone who has ever worked with me knows I can sometimes get obsessed with a new technology, with learning about it and experimenting with it.

Fast forward to 2025, I've been doing a lot of agentic coding using Claude Code but it was mostly basic usage that demanded a lot of steering. Oftentimes I would end up writing the code myself. However, I noticed a significant leap when Claude Opus 4.5 was released. I actually liked the code the model was producing and it made me want to push it further and come up with cool new ideas on what to build with it.

I wanted to create something more impressive to me than, say, a React/Three.js single-page animation. What task would be so complex that it would normally take a year or more of my spare time? That would be a true challenge! Perhaps something that would resonate with my peers in the multimedia space…? That's when I thought about an AV1 encoder. Written in Rust, of course, with no dependencies, and no unsafe code. It was just a gamble, but even though I made an effort setting up the task, I had little confidence the model would succeed. I then ended up with a working version of it</a> in less than 12 hours. 😳

</video>

This is me encoded with wav1c (initial version)</figcaption> </figure>

It's not the best AV1 encoder by any measure, nor the fastest. The point is: I was able to develop a usable AV1 encoder in less than a day. It is specification compliant and we can decode the resulting bitstream with dav1d</a>, hardware decoder via macOS VideoToolbox API and, potentially, by any other AV1 decoder (I haven't tested all of them myself. If you do, let me know). To me, this was an astonishing exercise, as writing video encoders (even a bad one), is not a "less than a day" kind of task. Even more so, when we're talking about a production video codec like AV1</a>.

I still need to reflect more on the implications of my exercise. Specification-based development seems to be a great fit for agentic coding with a self-verification loop, encode/decode. A lower bar for custom encoders/decoders could mean being able to create custom encoding profiles that go beyond just a change of parameters, e.g. having a custom transform or prediction logic baked into the encoder, as opposed to simply tweaking qp/rate-control knobs. Trimmed down encoders/decoders could also be used in embedded devices or embedded on a website. I created a simple demo</a> to show how wav1c</a> (Wondrous AV1 Coder) compiled to WASM</a> can encode to AV1 in real time, no server-side processing needed. I added some stats so that we can take a closer look at what is going on. Do not expect high quality images, but it absolutely works. You can download the MP4 file and play it in VLC or QuickTime (if you have AV1 hardware support as of M3 or more recent MacBook). Even without any practical applications, this code base can also be used simply for learning and to spark creativity.

</a>

The wav1c demo page, realtime AV1 encoding in the browser with WASM.</figcaption> </figure>

🥚🐰 Click me for a fun fact</summary>

This screenshot is an AVIF file encoded with wav1c itself.

./ffmpeg</span> -y -i</span> Screenshot.png</span> -c:v</span> libwav1c</span> -wav1c-q 1</span> demo-web-ui.avif</span></span></code></pre></details>
If you want to try locally, I also included a patch file</a> you can apply and use wav1c with FFmpeg.</p>
# Build the wav1c static library</span></span>
cargo</span> build</span> -p</span> wav1c-ffi</span> --release</span></span>
</span>
# Clone and patch FFmpeg</span></span>
git</span> clone https://git.ffmpeg.org/ffmpeg.git</span></span>
cd</span> ffmpeg</span></span>
git</span> apply /path/to/wav1c/ffmpeg-libwav1c.patch</span></span>
</span>
# Configure with libwav1c (adjust library path as needed)</span></span>
./configure</span> --enable-libwav1c</span> \</span></span>
  --extra-cflags=</span>"</span>-I/path/to/wav1c/wav1c-ffi/include</span>"</span> \</span></span>
  --extra-ldflags=</span>"</span>-L/path/to/wav1c/target/release</span>"</span></span>
</span>
make</span> -j$(</span>nproc</span>)</span></span>
</span>
# Encode</span></span>
./ffmpeg</span> -i</span> input.y4m</span> -c:v</span> libwav1c</span> -wav1c-q 64</span> output.mp4</span></span></code></pre>
You can embbed wav1c in your Rust project, or embedded microcontroller firmware. The Rust API is very straight forward to use:</p>
use</span> wav1c</span>::</span>{</span>Encoder</span>,</span> EncoderConfig</span>,</span> FrameType</span>};</span></span>
use</span> wav1c</span>::</span>y4m</span>::</span>FramePixels</span>;</span></span>
</span>
let</span> config</span> =</span> EncoderConfig</span> {</span></span>
    base_q_idx</span>:</span> 128</span>,</span></span>
    keyint</span>:</span> 25</span>,</span></span>
    target_bitrate</span>:</span> None</span>,</span></span>
    fps</span>:</span> 25.0</span>,</span></span>
};</span></span>
</span>
let mut</span> encoder</span> =</span> Encoder</span>::</span>new</span>(</span>320</span>,</span> 240</span>, config)</span>.</span>unwrap</span>();</span></span>
let</span> frame</span> =</span> FramePixels</span>::</span>solid</span>(</span>320</span>,</span> 240</span>,</span> 128</span>,</span> 128</span>,</span> 128</span>);</span></span>
</span>
encoder</span>.</span>send_frame</span>(</span>&</span>frame)</span>.</span>unwrap</span>();</span></span>
let</span> packet</span> =</span> encoder</span>.</span>receive_packet</span>()</span>.</span>unwrap</span>();</span></span>
</span>
assert_eq!</span>(packet</span>.</span>frame_type,</span> FrameType</span>::</span>Key</span>);</span></span>
// packet.data contains raw AV1 OBUs (TD + SequenceHeader + Frame)</span></span></code></pre>
It's a contentious but absolutely fascinating time to be a software engineer</a>. A friend told me last week, the LLM model I'm using today is the worst model I will ever use. I reckon he's right and that doesn't faze me in any way, it urges me to keep on top of it. I want to learn more and know how to better take advantage of these tools</em> in my work. Because these are tools, just like a code linter or cargo clippy</code> are. They don't replace my creativity, they allow me to take on more ambitious and increasingly complex challenges. I'm accountable for what I create and I care deeply about the result of my work, whether or not it's developed using Turbo Pascal, Vim, NeoVim, IntelliJ, VS Code, or Claude Code.</p>

Learn by example: Making it easier to understand GStreamer 2025-11-17T00:00:00+00:00 We like step-by-step learning. I have been using and learning GStreamer Recently, I thought about yet another way The Core GStreamer is an object oriented framework, Buffers</h3> Buffers hold the data we want to send through Clocks</h3> The next concept we gonna implement are Clocks. current wait_duration<&# jitter time.sleep(wait_ actual_time jitter </code></ The Clock class will help us to track progress Elements</h3> Elements are the steps of a GStreamer pipeline. old_state I tried to keep this as simple as possible. </d Elements always transition between </div> </section> Elements always transition between all the NULL : Initial state READY : Preparation PAUSED : This is when PLAYING : Indicates </ul> You can read in much more detail about each Lastly, we add a few methods to help us create Pads</h3> This might be a new concept for you, but Our Pad implementation in gst_mini</code> peer_pad.peer<&# </code></ One aspect of Pads that we must learn is The call stack of the chain function would encoder.chain() encoder.chain() segmenter.chain() segmenter.chain() publisher.chain() publisher.chain() segmenter.chain() encoder.chain() datasource - PUSH We will dive more on this process in the Pipeline</h3> Another central concept of GStreamer are element.pipe src.link(dst)<&# element.set_ start Segment</h3> The last core concept we will represent is running_time The following formula is how we transform $$ running_time = \frac{position - start}{rate} + base $$ The $position$ (or PTS) of the buffer is Why it is important to learn how to convert Creating We have a bare bones GStreamer mini implementation, The first element we will create is a "fake pts running_time clock_time<& _, jitter_ms<&# We implement the basic functionality of a We can now start creating a pipeline and Simple Manually demonstrating pipeline sink pipeline.add(sink)<& sink.segment pipeline.set_state(G pts buffer )<&# sink.sink_pad._c pipeline.set_state(G main()<& I highly encourage you to modify this code Now we can implement a camera like element, next_frame_time< current_time eos_buffer }<&# This is a slightly more complex element implementation, Now we have everything necessary to create pipeline source sink pipeline.add(source) pipeline.add(sink)<& source.link(sink)<&# pipeline.set_state(G pipeline.run( pipeline.set_state(G main()<& Look at the logs and you will notice all The fun continues</h3 We have also S3Sink, HLSSegmenter, VideoEnc, pipeline source queue segmenter sink pipeline.add(source, pipeline.link(source, pipeline.link(queue, pipeline.link(segmenter, pipeline.set_state(G pipeline.run( pipeline.set_state(G source_stats queue_stats main()<& </div> I highly encourage you to study the logs Can While playing around with this project I This example is almost a 1-1 usage of GStreamer, gi.require_version(gi.require_version( Gst.init( Gtk.init( pipeline filesrc pngdec videoconvert imagefreeze gtksink filesrc.set_property pipeline.add(filesrc pipeline.add(pngdec) pipeline.add(videoco pipeline.add(imagefr pipeline.add(gtksink filesrc.link(pngdec) pngdec.link(videocon videoconvert.link(im imagefreeze.link(gtk loop bus bus.add_signal_watch t loop.quit()< err, loop.quit()< bus.connect( loop.quit()< signal.signal(signal ret sys.exit( loop.run()</ pipeline.set_sta main()<& Wait! Is that GStreamer is a project that has been built No caps negotiation</li> No events (except implicit segment)</li> No queries</li> No buffer pools or memory management</li> No state change ordering</li> No preroll mechanism</li> Single clock (no clock providers/selection)< No activation modes (push/pull)</li> No scheduling/chain optimization</li> No latency calculation</li> etc</li> </ul> Many of the media services you use todayMy Late-Night Hack with iOS Shortcuts and AI 2025-01-11T00:00:00+00:00 multimedia framework for the last +5 years, during this time I have also sent back some contributions. My first contribution was the implementation of the Rust version of the HLS sink element in 2021 (hlssink3</code>). But I cannot say that I had completely grasped the framework concepts, it is a very large project and full of conventions. You can only learn those conventions by reading the GStreamer code, writing GStreamer applications, and participating in the community. Still I constantly find myself thinking about pads, elements, pipeline, etc, and learning something new. to deepen my knowledge of GStreamer: write GStreamer myself! Of course I am joking here, but why not a greatly simplified version of the framework focused on learning the basic concepts. Things like pads, elements, pipeline and buffers and some time/clock handling will get you a long way. I get asked frequently about "how to learn GStreamer" and my answer is always: write code with GStreamer. Today, we will write "GStreamer code" from scratch, so you won't feel overwhelmed by all the information. We will call it "GStreamer Mini" (or gst_mini</code>) and it is written in Python</a>. Concepts and Objects</h2> we have classes inheritance, properties, all the usual suspects (OOP abstractions). On top of those, like all code frameworks, we have core classes that we use to build upon. We will learn about Buffers, Pads, Elements, Pipeline, Clocks. the pipeline. Buffers can hold any kind of data, GStreamer does not impose any limits on that. Ideally, buffers hold data that is temporal and produced over time in a streaming way (thus the name "streamer"). You can add anything to buffers, people has been using it to hold coded multimedia data, but you can use it to hold readings from your thermostat (that could be a cool GStreamer element actually). Trimming buffers to their bare bones fields, we are left with presentation time (PTS), duration, flags, and data. #282A36;">from enum import Flag, auto from typing import Any class BufferFlags(Flag): """Buffer flags similar to GstBufferFlags.""" NONE = 0 DISCONT = auto() # Discontinuity in the stream EOS = auto() # End of stream marker DELTA_UNIT = auto() # Indicates the buffer is not decodable by itself class GstBuffer: """ Simplified GstBuffer carrying timestamped data. Corresponds to GstBuffer in gstbuffer.h:80-133 """ def __init__(self, pts: float, duration: float, data: Any, flags: BufferFlags = BufferFlags.NONE): """ Create a buffer. Args: pts: Presentation timestamp in seconds (like GST_BUFFER_PTS) duration: Buffer duration in seconds (like GST_BUFFER_DURATION) data: Payload data (dict for simulated frames, or segment info) flags: Buffer flags (DISCONT, EOS, etc.) """ italic;"> self.pts = pts italic;"> self.duration = duration italic;"> self.data = data italic;"> self.flags = flags def has_flag(self, flag: BufferFlags) -> bool: """Check if buffer has a specific flag.""" return bool(self.flags & flag) def set_flag(self, flag: BufferFlags): """Set a flag on the buffer.""" italic;"> self.flags |= flag def unset_flag(self, flag: BufferFlags): """Unset a flag on the buffer.""" italic;"> self.flags &= ~flag</code></pre> can set in our buffers. GStreamer has way more flags than those for buffers, but the EOS</code>, DELTA_UNIT</code>, and DISCONT</code> will get us a long way. Those flags are direct takes from the real GStreamer codebase. As a streaming over time framework clocks are a very important concept for timing and syncronization. GStreamer implements multiple types of clocks, the basic definition of a clock in GStreamer is that it is always increasing through time (never goes backwards) and produces an absolute time. We will use the Python's time</code> module monotonic</code> function and write a wrapper class around it with some helpers. #282A36;">import time from typing import Tuple class GstClockReturn: """Return values from clock operations.""" OK = "ok" EARLY = "early" UNSCHEDULED = "unscheduled" BADTIME = "badtime" class GstClock: """ Simplified GstClock for timing and synchronization. Corresponds to gst_clock_id_wait in gstbasesink.c:2381 """ def __init__(self): """Create a clock.""" italic;"> self._start_time = None def start(self): """Start the clock (called when pipeline goes to PLAYING).""" italic;"> self._start_time = time.monotonic() def get_time(self) -> float: """ Get current clock time in seconds. Returns: Time in seconds since clock started, or 0 if not started """ if self._start_time is None: return 0.0 return time.monotonic() - self._start_time def wait_until(self, target_time: float) -> Tuple[str, float]: """ Wait until the clock reaches the target time. This is similar to gst_clock_id_wait in gstbasesink.c:2381 Args: target_time: Target time to wait for (in seconds) Returns: Tuple of (GstClockReturn, jitter in seconds) - jitter > 0: woke up late - jitter < 0: woke up early - jitter = 0: perfect timing """ if self._start_time is None: return (GstClockReturn.BADTIME, 0.0) style="color: #FF79C6;"> = self.get_time() x2F;span> = target_time - current if wait_duration <= 0: # Already past target time - return immediately with negative jitter style="color: #FF79C6;"> = -wait_duration return (GstClockReturn.OK, jitter) # Sleep until target time duration) # Calculate jitter (how far off we were) style="color: #FF79C6;"> = self.get_time() style="color: #FF79C6;"> = actual_time - target_time return (GstClockReturn.OK, jitter) ;pre> of time after a start instant. It also has a wait_until</code> method which will be essential later for us to handle synchronisation in our elements. Again, those methods have direct relation to real GStreamer methods. Elements may produce, process, and receive buffers over time. Elements are specialised and focused on one function they process the buffers. I have talked about the different types of elements in GStreamer in my previous blog post "A brief introduction to GStreamer</a>". Here we will represent the most basic functionality of elements in GStreamer plus some helper methods. #282A36;">from abc import ABC, abstractmethod from enum import Enum, auto from typing import Optional from .pad import GstPad from .segment import GstSegment class GstState(Enum): """Element states (simplified version of GstState).""" NULL = auto() READY = auto() PAUSED = auto() PLAYING = auto() class GstElement(ABC): """ Base class for all pipeline elements. Elements are the processing units in a pipeline. """ def __init__(self, name: str): """ Create an element. Args: name: Element name """ italic;"> self.name = name italic;"> self.state = GstState.NULL italic;"> self.src_pad: Optional[GstPad] = None italic;"> self.sink_pad: Optional[GstPad] = None italic;"> self.segment = GstSegment() italic;"> self.pipeline: Optional['GstElement'] = None # Will be set by pipeline def log(self, message: str): """ Log a message with element context. Args: message: Message to log """ print(f"[{self.pipeline.clock.get_time():.3f}s] {self.name}: {message}") def create_src_pad(self, name: str = "src") -> GstPad: """ Create a source pad. Args: name: Pad name Returns: Created source pad """ italic;"> self.src_pad = GstPad(name, self) return self.src_pad def create_sink_pad(self, name: str = "sink") -> GstPad: """ Create a sink pad with chain function. Args: name: Pad name Returns: Created sink pad """ italic;"> self.sink_pad = GstPad(name, self) return self.sink_pad def link(self, downstream: 'GstElement'): """ Link this element to a downstream element. Args: downstream: Element to link to """ if self.src_pad is None: raise ValueError(f"{self.name} has no source pad") if downstream.sink_pad is None: raise ValueError(f"{downstream.name} has no sink pad") italic;"> self.src_pad.link(downstream.sink_pad) def set_state(self, state: GstState): """ Change element state. Args: state: New state """ if self.state == state: return style="color: #FF79C6;"> = self.state # Handle direct jumps (e.g., NULL -> PLAYING) # Go through intermediate states if old_state == GstState.NULL and state == GstState.PLAYING: italic;"> self.set_state(GstState.READY) italic;"> self.set_state(GstState.PAUSED) italic;"> self.set_state(GstState.PLAYING) return elif old_state == GstState.NULL and state == GstState.PAUSED: italic;"> self.set_state(GstState.READY) italic;"> self.set_state(GstState.PAUSED) return italic;"> self.state = state # Call subclass handlers if state == GstState.READY and old_state == GstState.NULL: italic;"> self.on_ready() elif state == GstState.PAUSED and old_state == GstState.READY: italic;"> self.on_paused() elif state == GstState.PLAYING and old_state == GstState.PAUSED: italic;"> self.on_playing() elif state == GstState.PAUSED and old_state == GstState.PLAYING: italic;"> self.on_paused_from_playing() elif state == GstState.NULL: italic;"> self.on_null() # State change hooks (can be overridden by subclasses) def on_ready(self): """Called when transitioning to READY state.""" pass def on_paused(self): """Called when transitioning to PAUSED state.""" pass def on_playing(self): """Called when transitioning to PLAYING state.""" pass def on_paused_from_playing(self): """Called when transitioning from PLAYING to PAUSED.""" pass def on_null(self): """Called when transitioning to NULL state.""" pass</code></pre> One important part of this code is the handling of transition of states of elements. aria-labelledby="bsoM+oFc"> iv> aria-hidden="true">Element State Transitions states in order: NULL → READY → PAUSED → PLAYING. This is critical and allows elements to configure themselves for pre-processing or reset their internal state. Never skip states! following states in order: of elements or you can think of it as an uninitialized state;</li> of the element;</li> the element is basically ready to start processing data;</li> the element is ready or already processing data.</li> of those states in the official documentation</a>. pads on our elements. Pads are the next concept we will dive into. you can think of pads like pipes where you push buffers between elements. You need to connect pads from a source element to a destination or sink element. Pads are another critical concept in GStreamer. Elements may contain one or many pads which can be dynamically created or static to the element (always present). is greatly simplified. But I have tried to keep it relatable to the actual Pad implementation in the GStreamer codebase. Pads have a peer pad, which is populated when the pad is "connected". A pad connection is a complex process in GStreamer, but the essence is that pads need to be compatible to connect. Pads have a "type", which is called "caps" or capabilities, defining what type of data the pad is capable of handling. We are not representing capabilities in our codebase. We will be responsible and only connect compatible pads in our examples, promise. #282A36;">import threading from typing import Callable, Optional, TYPE_CHECKING from enum import Enum, auto from .buffer import GstBuffer if TYPE_CHECKING: from .element import GstElement class GstFlowReturn(Enum): """Return values for pad operations (similar to GstFlowReturn).""" OK = auto() EOS = auto() FLUSHING = auto() ERROR = auto() class GstPad: """ Simplified GstPad for connecting elements. Corresponds to gst_pad_push / gst_pad_chain_data_unchecked in gstpad.c:4497-4586 """ def __init__(self, name: str, element: 'GstElement'): """ Create a pad. Args: name: Pad name (e.g., "src", "sink") element: Parent element """ italic;"> self.name = name italic;"> self.element = element italic;"> self.peer: Optional['GstPad'] = None italic;"> self.chain_function: Optional[Callable[[GstBuffer], GstFlowReturn]] = None # Thread safety - similar to GST_PAD_STREAM_LOCK italic;"> self._lock = threading.Lock() italic;"> self._flushing = False def link(self, peer_pad: 'GstPad'): """ Link this pad to a peer pad. Args: peer_pad: The pad to link to """ italic;"> self.peer = peer_pad x2F;span> = self def set_chain_function(self, func: Callable[[GstBuffer], GstFlowReturn]): """ Set the chain function for this pad. Args: func: Function to call when receiving buffers """ italic;"> self.chain_function = func def push(self, buffer: GstBuffer) -> GstFlowReturn: """ Push a buffer to the peer pad. This is similar to gst_pad_push() in gstpad.c:4795 Args: buffer: Buffer to push Returns: GstFlowReturn indicating success/failure """ if self.peer is None: return GstFlowReturn.ERROR # Call peer's chain function - this is the synchronous call # that makes chain calls blocking (gstpad.c:4560) return self.peer._chain(buffer) def _chain(self, buffer: GstBuffer) -> GstFlowReturn: """ Internal chain function handler. Similar to gst_pad_chain_data_unchecked in gstpad.c:4497 Args: buffer: Buffer received Returns: GstFlowReturn from the chain function """ # Acquire stream lock (GST_PAD_STREAM_LOCK at gstpad.c:4509) with self._lock: if self._flushing: return GstFlowReturn.FLUSHING if self.chain_function is None: return GstFlowReturn.ERROR # Call the element's chain function (gstpad.c:4560) return self.chain_function(buffer) ;pre> the "chain function". The chain function of a pad is the main way the buffer is handled by the pad and consequently by the element that the pad belongs to. When a source element pushes buffers to its src pad, the sink chain function of the peer pad is called. We implement this in our code and it is similar to what the GStreamer codebase does as well. look like something like this: #282A36;">datasource - PUSHING buffer 0 - RECEIVED buffer - PUSHING encoded buffer downstream - RECEIVED buffer - PUSHING segmented buffer downstream - RECEIVED buffer - RETURNING OK to caller - RETURNING OK to caller - RETURNING OK to caller RETURNED: OK</code></pre> future. pipelines. As you might have guessed, pipelines hold a collection of elements that must work in synchrony. Pipelines trigger operations on elements and hold the shared concept of time among its elements. Elements must be added to a pipeline to work properly in GStreamer. In most cases, your GStreamer based application will be built around a pipeline. Pipelines can be hold very complex graphs with many elements. #282A36;">import time from typing import List from .element import GstElement, GstState from .clock import GstClock class GstPipeline: """ Pipeline manages a collection of linked elements. Similar to GstPipeline in GStreamer. """ def __init__(self, name: str): """ Create a pipeline. Args: name: Pipeline name """ italic;"> self.name = name italic;"> self.elements: List[GstElement] = [] italic;"> self.clock = GstClock() italic;"> self.base_time = 0.0 italic;"> self.state = GstState.NULL def add(self, *elements: GstElement): """ Add elements to the pipeline. Args: *elements: Elements to add """ for element in elements: italic;"> self.elements.append(element) line = self def link(self, src: GstElement, dst: GstElement): """ Link two elements together. Args: src: Source element dst: Destination element """ x2F;span> def set_state(self, state: GstState): """ Set the state of all elements in the pipeline. Args: state: Target state """ print(f"[{self.clock.get_time():.3f}s] Pipeline: Setting state to {state.name}") # Special handling for PLAYING state if state == GstState.PLAYING and self.state != GstState.PLAYING: # Start the clock if self.clock._start_time is None: italic;"> self.clock.start() # Set base_time (corresponds to base_time in gstbasesink.c:2350) # In a real pipeline, this synchronizes multiple sinks italic;"> self.base_time = self.clock.get_time() print(f"[{self.clock.get_time():.3f}s] Pipeline: base_time set to {self.base_time:.3f}s") italic;"> self.state = state # Propagate state change to all elements for element in self.elements: state(state) def run(self, duration: float = 30.0): """ Run the pipeline for a specified duration. Args: duration: How long to run in seconds (default: 30s) """ print(f"[{self.clock.get_time():.3f}s] Pipeline: Running for {duration}s...") try: # Set to PLAYING if not already if self.state != GstState.PLAYING: italic;"> self.set_state(GstState.PLAYING) # Run for specified duration style="color: #FF79C6;"> = time.monotonic() while time.monotonic() - start < duration: time.sleep(0.1) except KeyboardInterrupt: print(f"\n[{self.clock.get_time():.3f}s] Pipeline: Interrupted by user") finally: # Clean shutdown print(f"[{self.clock.get_time():.3f}s] Pipeline: Stopping...") italic;"> self.set_state(GstState.NULL)</code></pre> functionalities. It allows us to add elements and to change the state of the pipeline which is then reflected to all its elements in order. The pipeline also hold the clock which is shared among all elements of that pipeline, this is essential to provide synchronous execution. When a pipeline starts the time is captured, and that becomes the base time of all processing. the concept of a GstSegment. Understand segments is important to decipher the time synchronisation in GStreamer. Segments allows us to convert the PTS of a buffer to its processing time based on the rate we want to process data. No matter if a pipeline process live content or pre-recorded content we want the processing to happen based on time. The realtime rate makes sense when processing live content or playing back the content, but if we are processing pre-recorded information we may want to process all the data of a 2 hours movie in an instant. GstSegment is an event in GStreamer, but we won't be talking about events in gst_mini for now. We introduce the concept of segment here to keep our code minimally relatable to the GStreamer codebase. #282A36;">class GstSegment: """ Simplified GstSegment for stream time to running time conversion. Corresponds to gst_segment_to_running_time in gstsegment.c:822-867 """ def __init__(self, start: float = 0.0, stop: float = None, rate: float = 1.0, base: float = 0.0): """ Create a segment. Args: start: Start time of the segment (in stream time) stop: Stop time of the segment (None = no limit) rate: Playback rate (1.0 = normal speed) base: Accumulated running time from previous segments """ italic;"> self.start = start italic;"> self.stop = stop italic;"> self.rate = rate italic;"> self.base = base def to_running_time(self, position: float) -> float: """ Convert stream time position to running time. Formula from gstsegment.c:822: running_time = (position - start) / rate + base Args: position: Position in stream time (e.g., buffer PTS) Returns: Running time, or -1 if position is outside segment boundaries """ # Check if position is within segment boundaries if position < self.start: return -1.0 if self.stop is not None and position > self.stop: return -1.0 # Convert: (position - start) / rate + base style="color: #FF79C6;"> = (position - self.start) / self.rate + self.base return running_time</code></pre> the buffer time in the relative running time of the pipeline: relative to the $start$ of the pipeline. Then this difference needs to be adjusted by the $rate$ of playback (which may be 1 in live or realtime pipelines). The $base$ is the base time of the segment, which can be 0 or more. A pipeline might produce many segments, which have a monotonically increasing base time which is the end of the previous segment. the buffer PTS to the running time? Because you will, at some point, find yourself in need to synchronise your element processing of a buffer to the global clock. Elements in gst_mini</h2> but how to create elements? In this section we will create a few elements and connect them together to see processing happening using our gst_mini</code> framework implementation. sink" element. This element does nothing, it just receive buffers and log information about those buffers. #282A36;">from ..core.element import GstElement from ..core.pad import GstFlowReturn from ..core.buffer import GstBuffer, BufferFlags class FakeSink(GstElement): def __init__(self, name, sync: bool = False): italic;"> super().__init__(name) # properties italic;"> self.sync = sync # state italic;"> self.buffer_count = 0 italic;"> self.sink_pad = self.create_sink_pad("sink") italic;"> self.sink_pad.set_chain_function(self._chain) def on_ready(self): italic;"> self.buffer_count = 0 def _chain(self, buffer: GstBuffer) -> GstFlowReturn: if buffer.has_flag(BufferFlags.EOS): italic;"> self.log("Received EOS") return GstFlowReturn.EOS # Extract timestamp style="color: #FF79C6;"> = buffer.pts # Convert stream time to running time (gst_segment_to_running_time) # Corresponds to gstbasesink.c:2207 style="color: #FF79C6;"> = self.segment.to_running_time(pts) if running_time < 0: italic;"> self.log("Buffer outside segment, dropping") return GstFlowReturn.OK # Synchronization (gst_base_sink_do_sync) if self.sync and self.state.name == 'PLAYING': # Calculate clock time: running_time + base_time # Corresponds to gstbasesink.c:2356 #x2F;span> = running_time + self.pipeline.base_time # Wait on clock (gst_base_sink_wait_clock at gstbasesink.c:2381) italic;"> self.log(f"Waiting for clock_time={clock_time:.3f}s...") jitter = self.pipeline.clock.wait_until(clock_time) x2F;span> = jitter * 1000 if jitter >= 0: italic;"> self.log(f"Clock wait complete (jitter: {jitter_ms:+.1f}ms)") else: italic;"> self.log(f"Buffer late (jitter: {jitter_ms:+.1f}ms)") italic;"> self.buffer_count += 1 italic;"> self.log(f"Processed {repr(buffer)}") return GstFlowReturn.OK</code></pre> sink element, but without any other logic. The most interesting part of this code is that we implement the clock synchronisation similar to what the real GStreamer code does. We use the segment to calculate the running time and then use the pipeline clock to wait for that butter PTS in the pipeline clock before proceeding with processing. This is how sink elements, like displays can show the video frames at the exact time they need to be displayed to users. using our new fakesink. This is already a good example of how the concepts and classes we introduced previously work together. #282A36;">#!/usr/bin/env python3 """ example showing buffer synchronization. creates buffers and pushes them through a sink, the clock wait mechanism. Pipeline: Manual buffers → FakeSink (with sync) """ from gst_mini import GstPipeline, GstState, GstBuffer, GstSegment, FakeSink def main(): print("=" * 80) print("Simple Synchronization Example") print("=" * 80) print() print("Manually pushing 5 buffers with increasing timestamps") print("Watch for clock waits demonstrating synchronization") print() print("=" * 80) print() # Create pipeline style="color: #FF79C6;"> = GstPipeline("simple-pipeline") # Create sink style="color: #FF79C6;"> = FakeSink('fakesink', sync=True) # Add to pipeline #x2F;span> # Set up segment style="color: #FF79C6;"> = GstSegment(start=0.0, rate=1.0, base=0.0) # Set to PLAYING stState.PLAYING) print(f"[{pipeline.clock.get_time():.3f}s] Pushing buffers manually...\n") # Create and push buffers with increasing timestamps for i in range(5): style="color: #FF79C6;"> = i * 2.0 # 2 seconds apart style="color: #FF79C6;"> = GstBuffer( italic;"> pts=pts, italic;"> duration=2.0, italic;"> data={'segment_num': i, 'buffers': range(60), 'duration': 2.0} x2F;span> print(f"[{pipeline.clock.get_time():.3f}s] Pushing buffer {i} with PTS={pts}s") hain(buffer) print() # Cleanup stState.NULL) print() print("=" * 80) print("Notice:") print(" • Each buffer waited until its timestamp matched clock time") print(" • running_time = PTS (since segment.start=0, segment.base=0)") print(" • clock_time = running_time + base_time") print(" • Synchronization formula: wait until clock >= clock_time") print("=" * 80) if __name__ == '__main__': #x2F;span></code></pre> yourself. An especially interesting part I liked to adjust was the segment $rate$, which directly represents the rate of processing or playback of the buffers we are producing (where $rate = 1.0$ is normal speed, $rate > 1.0$ is faster, and $rate < 1.0$ is slower). We have a lot of logging and you will notice the changes in the time of processing. I find it super cool to play around. producing buffers live. #282A36;">"""LiveSource - Simulates live camera generating frames.""" import threading import time from ..core.element import GstElement, GstState from ..core.buffer import GstBuffer, BufferFlags from ..core.pad import GstFlowReturn class LiveSource(GstElement): """ Source element that generates frames at a fixed rate (like a camera). Demonstrates live sources that cannot be paused - they keep generating data. """ def __init__(self, name: str, fps: int = 30): """ Create a live source. Args: name: Element name fps: Frames per second to generate """ italic;"> super().__init__(name) italic;"> self.fps = fps italic;"> self.frame_interval = 1.0 / fps # Time between frames italic;"> self.frame_count = 0 italic;"> self.dropped_frames = 0 italic;"> self._thread = None italic;"> self._running = False italic;"> self._start_time = None # Create source pad italic;"> self.src_pad = self.create_src_pad("src") def on_playing(self): """Start generating frames when entering PLAYING state.""" italic;"> self.log(f"Starting frame generation at {self.fps} fps") italic;"> self._running = True italic;"> self._start_time = time.monotonic() italic;"> self._thread = threading.Thread(target=self._generate_frames, daemon=True) italic;"> self._thread.start() def on_null(self): """Stop generating frames when entering NULL state.""" italic;"> self.log("Stopping frame generation") italic;"> self.log(f"Generated {self.frame_count} frames, dropped {self.dropped_frames} frames") italic;"> self._running = False if self._thread is not None: italic;"> self._thread.join(timeout=1.0) def _generate_frames(self): """ Frame generation loop (runs in separate thread). This simulates a live camera that captures frames at regular intervals. """ /span> = time.monotonic() while self._running: = time.monotonic() # Check if it's time for the next frame if current_time >= next_frame_time: # Calculate PTS (presentation timestamp) # This is the time since we started, in seconds pts = current_time - self._start_time duration = self.frame_interval # Create buffer with frame data buffer = GstBuffer( italic;"> pts=pts, italic;"> duration=duration, italic;"> data={ 'frame': self.frame_count, 'timestamp': pts, 'content': f'frame_{self.frame_count:06d}' } ) # Try to push buffer downstream # This call may block if downstream is slow or queue is full result = self.src_pad.push(buffer) if result == GstFlowReturn.OK: if self.frame_count % self.fps == 0: # Log every second italic;"> self.log(f"Generated frame {self.frame_count}, PTS={pts:.3f}s") italic;"> self.frame_count += 1 else: # Push failed (queue full, flushing, etc.) italic;"> self.dropped_frames += 1 if self.dropped_frames % 10 == 1: italic;"> self.log(f"Dropped frame {self.frame_count} (total dropped: {self.dropped_frames})") # Schedule next frame next_frame_time += self.frame_interval else: # Sleep until next frame time sleep_time = next_frame_time - current_time if sleep_time > 0: time.sleep(min(sleep_time, 0.001)) # Sleep in small increments # Send EOS when stopping style="color: #FF79C6;"> = GstBuffer(pts=0.0, duration=0.0, data={}, flags=BufferFlags.EOS) italic;"> self.src_pad.push(eos_buffer) def get_stats(self) -> dict: """Get statistics about frame generation.""" return { 'frames_generated': self.frame_count, 'frames_dropped': self.dropped_frames, 'fps': self.fps, x2F;span></code></pre> but all the complexity is just to simulate a device producing content as specific time. We still leverage all the gst_mini framework and publish buffers to pads just like in real GStreamer code. our first full gst_mini pipeline and play it out. #282A36;">from gst_mini import GstPipeline, GstState, FakeSink, LiveSource def main(): print("=" * 80) print("Simple Synchronization Example") print("=" * 80) print() print("Using LiveSource to generate buffers at 1fps") print("Watch for clock waits demonstrating synchronization") print() print("=" * 80) print() # Create pipeline style="color: #FF79C6;"> = GstPipeline("simple-pipeline") # Create elements style="color: #FF79C6;"> = LiveSource('videotestsrc', fps=1) style="color: #FF79C6;"> = FakeSink('fakesink', sync=True) # Add to pipeline #x2F;span> # Link elements x2F;span> # Set to PLAYING stState.PLAYING) # Runs for 5 seconds style="color: #BD93F9;">5) # Cleanup stState.NULL) print() print("=" * 80) print(f"Completed example, total processed frames by sink: {sink.buffer_count}") print("=" * 80) if __name__ == '__main__': #x2F;span></code></pre> the processing was right on time. I have also created a few other elements we can play around. > and Queue. Those extra elements can be combined to create more interesting pipelines for education. This is not dealing with real multimedia content, this is just for demostration. Still I tried to follow some of the GStreamer concepts with them and may use them in future blog posts. For now, here we can see a more complex example combining some of those extra elements: background-color: #282A36;">from gst_mini import GstPipeline, GstState, LiveSource, Queue, HLSSegmenter, S3Sink def main(): print("=" * 80) print("Full HLS Pipeline Example") print("=" * 80) print() print("Pipeline: LiveSource → Queue → HLSSegmenter → S3Sink") print("- LiveSource generates 30 frames/sec") print("- Queue decouples threads (max 10 buffers, leaky upstream)") print("- HLSSegmenter creates 6-second segments") print("- S3Sink uploads with synchronization") print() print("Watch for:") print(" • Frame generation at steady rate (Thread A)") print(" • Queue filling/emptying") print(" • Segment creation every 6 seconds") print(" • Clock waits before upload (Thread B)") print(" • No blocking of frame generation!") print() print("=" * 80) print() # Create pipeline style="color: #FF79C6;"> = GstPipeline("hls-pipeline") # Create elements style="color: #FF79C6;"> = LiveSource("camera", fps=30) style="color: #FF79C6;"> = Queue("queue", max_size=10, leaky="upstream") style="color: #FF79C6;"> = HLSSegmenter("segmenter", target_duration=6.0) style="color: #FF79C6;"> = S3Sink("s3sink", bucket="live-streams", sync=True) # Add to pipeline queue, segmenter, sink) # Link elements queue) segmenter) sink) stState.PLAYING) # Run for 20 seconds style="color: #FFB86C;font-style: italic;">duration=20.0) stState.NULL) # Print statistics print() print("=" * 80) print("Statistics:") print("-" * 80) style="color: #FF79C6;"> = source.get_stats() print(f"LiveSource: {source_stats['frames_generated']} frames generated, " f"{source_stats['frames_dropped']} dropped") style="color: #FF79C6;"> = queue.get_stats() print(f"Queue: {queue_stats['buffers_in']} in, {queue_stats['buffers_out']} out, " f"{queue_stats['buffers_dropped']} dropped") print(f"S3Sink: {sink.segment_count} segments uploaded") print("=" * 80) if __name__ == '__main__': #x2F;span></code></pre> and add more logs in critical parts like state changes, pad handling buffer processing. You will create a very useful mental model that will help you read real GStreamer element code and also help you create your own elements. I like to browse through the elements in the gst-plugins-rs repository a lot. I really enjoy Rust and find the Rust bindings to be really good and readable, also using Rust for writing elements will prevent some common mistakes as the Rust type system is leveraged greatly to guide you. GstMini do multimedia? Why not?</h2> decided to implement a small multimedia example. For that I implemented a TkSink</code> for displaying frame bytes to a window canvas. I used Tcl/Tk GUI toolkit just because it was the lowest effort as it comes together with the Python installation (usually). Also a very simplified FileSrc</code> element but that is essentially similar to the real filesrc</code> element. You can look up the example</a> here: and you can even run this real GStreamer pipeline to replicate it: #282A36;">gst-launch-1.0 filesrc location=$HOME/Downloads/simple.png ! pngdec ! videoconvert ! imagefreeze ! osxvideosink</code></pre> Python: #282A36;">#!/usr/bin/env python3 import gi import sys import signal style="color: #E9F284;">'Gst', '1.0') style="color: #E9F284;">'Gtk', '3.0') from gi.repository import Gst, GLib, Gtk def main(): style="color: #BD93F9;">None) style="color: #BD93F9;">None) style="color: #FF79C6;"> = Gst.Pipeline.new("test-pipeline") style="color: #FF79C6;"> = Gst.ElementFactory.make("filesrc") style="color: #FF79C6;"> = Gst.ElementFactory.make("pngdec") style="color: #FF79C6;"> = Gst.ElementFactory.make("videoconvert") style="color: #FF79C6;"> = Gst.ElementFactory.make("imagefreeze") style="color: #FF79C6;"> = Gst.ElementFactory.make("autovideosink") ("location", "/media/simple.png") ) nvert) eeze) ) vert) agefreeze) sink) # Create GLib MainLoop style="color: #FF79C6;"> = GLib.MainLoop() # Handle bus messages style="color: #FF79C6;"> = pipeline.get_bus() () def on_message(bus, message): style="color: #FF79C6;"> = message.type if t == Gst.MessageType.EOS: print("End of stream") /span> elif t == Gst.MessageType.ERROR: debug = message.parse_error() print(f"ERROR: {err.message}") if debug: print(f"Debug info: {debug}") /span> elif t == Gst.MessageType.STATE_CHANGED: if message.src == pipeline: old_state, new_state, pending_state = message.parse_state_changed() print(f"Pipeline state changed from {old_state.value_nick} to {new_state.value_nick}") return True style="color: #E9F284;">"message", on_message) # Handle SIGINT (Ctrl+C) gracefully def signal_handler(sig, frame): print("\nInterrupted by user") F;span> .SIGINT, signal_handler) # Set pipeline to PLAYING state style="color: #FF79C6;"> = pipeline.set_state(Gst.State.PLAYING) if ret == Gst.StateChangeReturn.FAILURE: print("ERROR: Unable to set the pipeline to the playing state.") style="color: #BD93F9;">1) # Run the main loop try: print("Pipeline is playing - press Ctrl+C to stop") ;span> except KeyboardInterrupt: print("\nInterrupted by user") finally: # Clean shutdown te(Gst.State.NULL) print("Test completed.") if __name__ == '__main__': #x2F;span></code></pre> GStreamer?</h2> over 20 years by very experienced engineers dealing with multimedia processing. The simplification in gst_mini does not reflect in any way the amount of details and pitfals GStreamer prevents or takes care for you. A non-exhaustive list of features we did not cover in GstMini: /li> may rely on GStreamer one way or another. Do not be fooled by the ugly GStreamer website! I hope this post was helpful to you and that it was inspiring for you to continue learning more about GStreamer. You know those moments right before falling asleep when your mind wanders and suddenly you get an idea you just have to try out? That's exactly what happened to me last night. Living in Amsterdam as a non-Dutch speaker, I've struggled to keep up with local news and understand the cultural context behind the stories. Sure, I could use Google Translate, but I wanted something more… comprehensive. The problem</h2> As an immigrant in The Netherlands, I face two main challenges: The language barrier (I am still studying Dutch, but nowhere near being proficient enough to read a whole article)</li> Understanding the cultural context behind news stories</li> </ol> The spark</h2> I remembered that iOS has this feature called Shortcuts that lets you create automations by connecting different app actions together. What caught my attention was that both Claude and ChatGPT have APIs that can be accessed through Shortcuts. This got me thinking – what if I could create a simple tool to help me better understand Dutch news articles? The Solution: A 10-Minute Hack</h2> Building upon an existing shortcut I had for removing ads from web pages, I put together a multi-step workflow that does the following: Takes a Safari webpage as input</li> Cleans up the content (removes ads, navigation, footers)</li> Translates the article from Dutch to English</li> Provides additional context like: Cultural references explained for foreigners</li> Sentiment analysis</li> How a native Dutch person might react to the topic</li> A concise summary of the main points</li> </ul> </li> </ol> The best part? It took only about 10 minutes to put together, all done on my phone! The Technical Bits</h2> What makes this interesting from an engineering perspective is the multi-step prompt technique I used. The workflow is split into two main steps: First prompt: Clean up and extract only the relevant article content Removes all the noise (text from overlays, menus, footers, etc)</li> Extracts just the article title and body</li> Outputs clean content</li> </ul> </li> Second prompt: Process and enhance the content Translates the clean content to English</li> Adds cultural context</li> Provides additional insights</li> </ul> </li> </ol> The result gets saved directly to the Notes app for easy access and historical view. Why Multi-Step Prompting?</h2> You might wonder why I split this into multiple steps instead of doing everything in one go. The answer lies in the "garbage in, garbage out" principle. By first cleaning up the input text, we ensure that the AI model only works with relevant content, leading to better quality outputs. Why This Matters</h2> Now, this isn't about becoming a "prompt engineer" – that's not the point. As software engineers, we're constantly looking for tools that can increase our productivity. Just like we use linters to catch known issues in our code, or how we rely on helpful compiler messages from Rust or Python to guide us through error resolution, I see this as just yet another tool. The real value here isn't in the prompts themselves, but in identifying opportunities where AI can help solve real-world problems we face. In my case, it was about breaking down the language barrier and cultural gap I experience living in the Netherlands. Try It Yourself</h2> I've recorded a quick demo of the tool in action. You can install the shortcut and try it yourself by using the Claude version of the Shortcut</a> or the ChatGPT version</a>. Feel free to modify the prompts for your own needs with different languages and countries. https://youtube.com/shorts/Qdv4fHdBfOw?si=io3TeGQurtY0IYyE </div> Prerequisites You'll need to have either Claude or ChatGPT installed on your iOS device to use this tool. </div> </section> Sometimes the best solutions come from scratching your own itch, even if it's right before bedtime. This is not my first time, and I don't expect it to be the last. 😴 Have you built any interesting shortcuts or tools to help with in your everyday life and leveraged LLMs for it? I'd love to hear about them, you can find me on BlueSky</a>, the Fediverse</a>, LinkedIn</a> or GitHub</a>. Readable Rust: Combinators vs. Match Clauses 2023-03-18T00:00:00+00:00 When writing code in any programming language, choosing the appropriate constructs is crucial for creating readable and maintainable code, and in Rust is no different. In this post, I will discuss a specific dilemma I find myself thinking often about: deciding between using combinators or match clauses. Let's consider an example where in an application we need to retrieve the value of an environment variable, convert it to a map, and then serialize it to JSON. The first approach uses combinators and closures: std::env::var("SELECTOR") .ok() .and_then(|env_var| selector_field(env_var.as_str()).ok()) .map(|sel_map| serde_json::to_value(sel_map).unwrap())</code></pre> I ignore the declaration of the fn selector_field(&str) -> Option<BTreeMap<String, String>></code> function for simplification. This version is concise and elegant, but doesn't log any warnings if and for what reason the result turns out to be None</code>. To add some context and make it easier for debugging, we can log some useful information. We can modify the code like this: std::env::var("SELECTOR") .map_err(|err| { log::warn!("Failed to find environment configuration for selector: {}", err); }) .ok() .and_then(|env_var| { selector_field(env_var.as_str()) .map_err(|err| { log::warn!("Failed to parse selector: {}", err); }) .ok() }) .map(|sel_map| serde_json::to_value(sel_map).unwrap())</code></pre> While functional, this version feels convoluted and harder to read due to the nested closures. In such cases, I tend to use a match clause, as I think it is a better option: let env_var = match std::env::var("SELECTOR") { Ok(env_var) => env_var, Err(err) => { log::warn!("Failed to find environment configuration for selector: {}", err); return None; } }; let sel_map = match selector_field(env_var.as_str()) { Ok(sel_map) => sel_map, Err(err) => { log::warn!("Failed to parse selector: {}", err); return None; } }; Some(serde_json::to_value(sel_map).unwrap())</code></pre> This version is more readable and easier to understand. Thus, I think it's useful to consider the following guidelines when choosing between combinators and match clauses: If you need to add multiple lines of code inside the closures when using combinators, a match clause might be a better choice.</li> When error handling is minimal or not needed (conversion to Option</code> or use of ?</code> is possible) and simple closures can be used in combinators, combinators are a good choice.</li> </ol> Combinators in Rust are a powerful tool for writing concise and functional code, but sometimes a match clause is still the better choice for readability. When deciding between the two, consider the complexity of error handling and the need for multiple lines of code inside closures used in combinators. By striking the right balance, we can create more readable and maintainable Rust code. Decompress content using Rust and flate2 without headaches 2023-01-26T00:00:00+00:00 A few days ago I had to deflate some compressed content using Rust. I quickly found the flate2</a> crate, it is well maintained and has a very high usage. I decided it would be a great fit for my little project, a CLI tool to download, decompress, and store in a local SQLite3 some content from AWS S3. I wrote my small CLI tool, and it seemed to work perfectly on the first run. It was all nice until some time later. I have noticed that some lines of my compressed file were missing. The code was running fine and didn't provide any warnings. After a few hours of debugging, I was sure there was a bug in my code somewhere. I went on and wrote a bunch more test to try to isolate in a reproducible way, where the problem was present, without any luck. All tests passing and no indication of problems anywhere. At some point, I decided to use a sample from the original compressed files, and that is how I managed to reproduce the bug. The sample code in the README of the flate2 crate was what I used in my code: use std::io::prelude::*; use flate2::read::GzDecoder; fn main() { let mut d = GzDecoder::new("...".as_bytes()); let mut s = String::new(); d.read_to_string(&mut s).unwrap(); println!("{}", s); }</code></pre> Unfortunately, the GzDecoder</code> cannot successfully decompress my files. I went then to the official library repository and found this issue</a> which describes exactly my problem. </div> Solution: Use MultiGzDecoder If GzDecoder fails to decompress your files silently, use MultiGzDecoder instead. It handles multiple gzip members in a single stream, which is common in concatenated gzip files. </div> </section> I had to use the MultiGzDecoder</code> instead, and it all worked as expected after this change. The example in the README of flate2 should probably be this: use std::io::prelude::*; use flate2::read::GzDecoder; fn main() { let mut d = MultiGzDecoder::new("...".as_bytes()); let mut s = String::new(); d.read_to_string(&mut s).unwrap(); println!("{}", s); }</code></pre> Finally, I still don't know why the MultiGzDecoder</code> works when the GzDecoder</code> don't. If you know, I will update this blog with your answer. Blogroll 2022-11-01T00:00:00+00:00 A curated collection of blogs and podcasts from creators whose content I regularly enjoy. Blogs</h2> Netflix Techblog</a> - Learn what my co-workers and I are working on</li> Kevquirk</a> - Provides consistently high-quality material across various topics</li> Stephen Diehl</a> - Features commentary examining digital finance and contemporary scams</li> </ul> Podcasts</h2> The Pragmatic Engineer</a> - Relevant content for sowftware engineers or anyone interested in working in tech.</li> </ul> About Me 2022-10-29T00:00:00+00:00 "I'm Rafael, really like technology and programming. This is my personal blog where I post anything that comes to mind." </blockquote> I'm a technology enthusiast and programmer. This is my personal blog where I share solutions to everyday technical problems and topics that interest me. I write about solutions discovered through my daily work and development activities. Topics I Write About</h2> I explore content related to: General programming concepts</li> Multimedia technology</li> Open Source and FOSS initiatives</li> Mechanical keyboards</li> Technology in general</li> </ul> Credits</h2> This blog is powered by Zola</a>. A brief introduction to GStreamer 2022-10-29T00:00:00+00:00 A while ago, I started working with GStreamer</a>. I wasn't familiar with the framework before, and took me a while to grasp how it works. I have recently created a mental model which has been helping me understand how to use GStreamer API and I thought sharing it here could also help others. GStreamer is a multimedia framework. It is meant to be used to create applications that can manipulate multimedia content. We don't usually manipulate the multimedia content in GStreamer directly, we orchestrate elements that deal with the content. GStreamer provides us with an API for orchestrating those elements. There are various types of elements: from audio and video encoding, decoding, demuxing, muxing to closed captions, image compositions and all kinds of metadata. There are over 90 elements</a> available in the official GStreamer distribution. It is likely that you'll find elements for most, if not all, the operations you might want to do. Element types</h2> When working with GStreamer you start with building a pipeline and using some elements to process your content. Building a pipeline consists generally of creating and connecting elements you need in a meaningful way. For example, to play a file, we need to read its content then decode the content to raw audio and send the raw audio to an audio device. In GStreamer there are a few general types of elements which can be categorized according to the way they connect with each other. Elements connect using a single or multiple pads which provide input and output streams of content. Input pads are called sink pads, and output pads are called source pads. Source elements</h3> Source elements have a single source pad always available. Read from external resources or produce content themselves. Source elements have one output stream or pad (source pad) available at all times (known as static pad). They do not accept any content as input (no sink pads). Destination elements</h3> Sink elements have a single sink pad always available. Destination elements, also known as sinks, are elements that receive a single stream of content. They don't produce any content out (no source pads). They usually have a single input (sink) pad available at all times (static pad). Demuxer elements</h3> Demuxer elements have a single sink pad always available and possibly multiple source pads created later on demand. Demuxer elements usually receive one stream of content and split the content into multiple output streams. They usually have a single input (sink) pad available at all times and will create new output (source) pads as soon as they figure out what types of content are available in the stream it's receiving. Muxer elements</h3> Muxer elements have a single source pad always available and multiple sink pads created later on demand. Muxer type elements receive multiple streams (sink pads) of content and mix them together into a single output stream (source pad). Muxers usually have an output (source) pad available at all times. Whenever we want a muxer to process some content, we need to request new input (sink) pads to be created. Filter elements</h3> Filter elements have a sink and a source pad always available. Filter elements generally have a single input (sink) pad and a single output (source) stream pad. The pads in a filter element are usually static (always available). Multimedia manipulation using pseudocode</h2> You now know about elements and their most common types. How to put them together? In GStreamer, elements usually connect inside a container (or bin) called a pipeline. We put elements into it and use API calls to make the elements connect. Let's say we want to play music from a file. We need to read the file, decode the content to raw audio, then send this content to an audio device. Here's a pseudo pipeline with made-up elements: Inspired by Javascript and HTML manipulation, one could imagine pseudocode like this: // Create our pipeline container let pipeline = createElement("pipeline"); // Create our elements let file_reader = createElement("filesource"); file_reader.setAttribute("location", "/Users/rafaelcaricio/astronaut.wav"); // Generic decoder which is similar // to a demuxer type of element let decoder = createElement("decoder"); let audio_device = createElement("audiodevice"); // Add all elements to the Pipeline container pipeline.appendChild(file_reader); pipeline.appendChild(decoder); pipeline.appendChild(audio_device); // Tell our elements how to connect file_reader.link(decoder); // The demuxer type of elements only create pads // after it is able to get some content, so we // add a event listener here to link later on // whenever the new stream pad is available let audio_device_pad = audio_device.getDestinationPad(); decoder.addEventListener("streamFound", function (stream_source_pad) { stream_source_pad.link(audio_device_pad); }) pipeline.dispatchEvent("playing"); pipeline.waitUntilEnd(); pipeline.dispatchEvent("stop");</code></pre> In this pseudocode, we are creating elements and manipulating them to connect together inside a pipeline container. If you have ever worked with web development, this should look familiar. This is comparable with how you will be thinking when working with GStreamer. Using the GStreamer API</h2> Let's translate this example to real GStreamer code to play a WAV file. We need to select our elements for every operation we want to do: read from a file, decode the content, send to an audio device. For those, we have filesrc</code>, decodebin</code>, autoaudiosink</code> respectively. [filesrc](https://gstreamer.freedesktop.org/documentation/coreelements/filesrc.html?gi-language=c#filesrc-page)</code>: Reads content from a file and has one output pad.</li> [decodebin](https://gstreamer.freedesktop.org/documentation/playback/decodebin.html?gi-language=c#decodebin-page)</code>: An abstract element that identifies the content received, and puts together other elements to decode the content and then create new pads with the types of content which are available.</li> [autoaudiosink](https://gstreamer.freedesktop.org/documentation/autodetect/autoaudiosink.html?gi-language=c)</code>: Selects the first available audio output device and sends the received audio content to be played.</li> </ul> Using GStreamer API in Rust</a> it looks like this: fn main() { gst::init().unwrap(); // Create our pipeline container let pipeline = gst::Pipeline::default(); let file_reader = gst::ElementFactory::make("filesrc").build().unwrap(); file_reader.set_property("location", "/Users/rafaelcaricio/astronaut.wav"); let demuxer = gst::ElementFactory::make("decodebin").build().unwrap(); let audio_device = gst::ElementFactory::make("autoaudiosink").build().unwrap(); pipeline.add(&file_reader).unwrap(); pipeline.add(&demuxer).unwrap(); pipeline.add(&audio_device).unwrap(); file_reader.link(&demuxer).unwrap(); // Our event listener to connect pads later on let audio_device_pad = audio_device.static_pad("sink").unwrap(); demuxer.connect_pad_added(move |_, pad| { pad.link(&audio_device_pad).unwrap(); }); pipeline.set_state(gst::State::Playing).unwrap(); // Wait until end, handle errors, etc let bus = pipeline.bus().unwrap(); for msg in bus.iter() { match msg.view() { MessageView::Eos(..) => break, MessageView::Error(err) => { println!("Oh no.. something wrong happened: {:?}", err); break } _ => continue, } } pipeline.set_state(gst::State::Null).unwrap(); }</code></pre> </div> As you can see, the fake JavaScript API we created for manipulating elements (like they would be HTML elements) and the real GStreamer API are analogous. The GStreamer API is vast and provides many operations to manipulate elements inside pipelines or inside other elements (also called bins). There are elements for many different operations you might want to do, not only conversion of content types. Here I generated a visualization of the GStreamer pipeline we wrote: </a> I like to think of elements as functionalities that we can piece together like LEGO blocks. If a source pad is of the same type as the sink pad, then they can connect. Pad types are called capabilities (or simply "caps") in GStreamer and are compatible when they have compatible fields. Caps look very much like mime-types</a> that we see in the HTTP header Content-Type</code>. An example of caps is audio/x-wav</code> which means an audio content in WAV format or video/x-raw,format=RGBA,width=480,height=320,framerate=30/1</code> which - you guessed it - is a decoded raw video content in RGBA format with the sizes and frame rate specified. Conclusion</h2> I hope this brief introduction has given you a good high-level overview of basic concepts that might help you jump start writing your own GStreamer multimedia processing pipelines. I recommend checking out the GStreamer documentation and browsing a little through some of the elements</a>, something might catch your eye. There is extensive documentation</a> and tutorials available in C</a> and Rust</a> that delve further into the concepts I introduced in this blog post. I've also created a basic GStreamer project template</a> that you can use. The GStreamer community is very friendly and welcoming to newcomers. We usually hang around at the official GStreamer IRC chat</a> which can be accessed using the matrix.org bridge</a>. Feel free to say "hi". See you around! Install GStreamer SRT plugin on macOS 2022-06-27T00:00:00+00:00 Today I was working on a project that uses GStreamer</a> and reads content using the SRT protocol</a> on my macOS. The SRT elements</a> could not be found in my system after installing GStreamer using Homebrew</a>. I’ve installed all the GStreamer sub-packages from Homebrew. $ brew install gstreamer gst-plugins-base \ gst-plugins-good gst-plugins-bad \ gst-plugins-ugly gst-libav \ gst-rtsp-server gst-editing-services</code></pre> Still, I could not find the strsrc</code> element, for example. $ gst-launch-1.0 -v srtsrc uri="srt://127.0.0.1:7001" ! fakesink WARNING: erroneous pipeline: no element "srtsrc"</code></pre> For some reason, not clear to me at the time, I did not have the plugin installed. This triggered me to look at the source code of GStreamer to find how the element is enabled. I know that GStreamer contains many plugins that depend on different third-party libraries. The SRT set of elements, resides in the gst-plugins-bad</code></a> bundle. Then it was clear to me that the SRT elements are only compiled if the libsrt</a> is available in the host system at compilation time. GStreamer meson file showing that it only enables the SRT element if the dependency is found. Ok, now I know what might be causing the SRT plugin to not be available on my GStreamer’s brew installation. In order to confirm that, I checked the Homebrew formula for its dependencies. </a> As I was guessing, libsrt is not a dependency of that formula. This means that the meson configuration we saw earlier is not letting the SRT plugin in the compilation process. The fix</h2> We need to modify the gst-plugins-bad</code> formula locally and then install from source. First, we uninstall the gst-plugins-bad</code> formula. $ brew rm gst-plugins-bad</code></pre> Then we can edit our local formula to add the libsrt</code> dependency. This is not strictly required, as we could just install libsrt manually and then recompile from source. But we will add here anyway, so we make sure we will not override this next time the package update. The following command will open your default editor as defined in the EDITOR</code> environment variable. $ brew edit gst-plugins-bad</code></pre> Add the line below to the dependency list: depends_on "srt"</code></pre> We now install the package from source. $ brew reinstall --build-from-source gst-plugins-bad</code></pre> That's it. Now you should have the SRT plugin installed and all its elements will be available in your system. We can double-check that by inspecting the one element, like the srtsrc</code>. $ gst-inspect-1.0 srtsrc</code></pre> You should see something like: Yay! That is it. Now I can continue my work and have all the SRT elements available on my macOS GStreamer installation from Homebrew. Why not just use async Python for this API? 2021-07-08T00:00:00+00:00 I quite frequently stumble upon people in the Python community being misled to think that using async Python code will make their APIs “run faster”. Async Python is a great feature and should be used with care. One point that I constantly find being overseen is the mix of sync and async code. The general rule is that we should never mix blocking code with async code. I would like to present in this post a simplified example where we can observe the usage of async Python will hurt the performance of an API and then see how we can fix it. Our example application is a FastAPI service that needs to call two operations from an external API within the handling of an HTTP request. Those are all the dependencies we will use for the example: # file requirements.txt fastapi[all]==0.65.1 uvicorn[standard]==0.13.4 requests==2.25.1 httpx==0.18.2</code></pre> Let's look at the example API code: # file app/application.py from fastapi import FastAPI import requests import uuid import logging logging.basicConfig(format="%(asctime)s %(message)s") log = logging.getLogger("myapp") log.setLevel(logging.DEBUG) app = FastAPI() EXTERNAL_API_ENDPOINT = "http://localhost:8888" @app.get("/healthcheck") async def healthcheck(): return {"status": "ok"} # # Async mixed with blocking # def internal_signing_op(op_num: int, request_id: str) -> None: session = requests.Session() response = session.request("GET", EXTERNAL_API_ENDPOINT, timeout=2000) print(f"{request_id} {op_num}: {response}") def sign_op1(request_id: str) -> None: internal_signing_op(1, request_id) def sign_op2(request_id: str) -> None: internal_signing_op(2, request_id) @app.get("/async-blocking") async def root(): request_id = str(uuid.uuid4()) print(f"{request_id}: started processing") sign_op1(request_id) sign_op2(request_id) print(f"{request_id}: finished!") return {"message": "hello world"}</code></pre> Here we have a simple application that tries to replicate the behavior that I'm trying to point out. We have mixed async code with the synchronous library requests</a>. The code works fine, but there is one problem. To understand the problem, we need to recap on how Uvicorn works. Uvicorn executes our application server by spawning workers</a> (OS sub-process) that handles the requests coming into our server. Every worker (sub-process) is a fully-featured CPython instance and has its own I/O loop</a> that runs our FastAPI application. The Main Process holds a socket that is shared with the workers and accepts the HTTP requests that are handled by the workers to actually process the request. We can have as many workers as we want, usually the number of CPU cores. In our case, to make it easier to analyze the behavior, we are going to run only a single worker. We execute our server with the following command: uvicorn app.application:app --workers 1</code></pre> I've set up a fake external API that we will use for this example. Just a simple server that takes a long time to execute some obscure operation (sleep(20)</code> 😄 ). # file external_api.py import asyncio from fastapi import FastAPI app = FastAPI() @app.get("/") async def root(): await asyncio.sleep(20) return {"message": "Hello World"}</code></pre> We spin up the external API server using this command: uvicorn external_api:app --port 8888 --workers 1</code></pre> We set 1 worker here for no good reason, the important part here is to make the external API run in the port 8888</code> which is the one we've hardcoded in our example application. Full working tree of the example for reference: . ├── app │ ├── __init__.py │ └── application.py ├── external_api.py └── requirements.txt 1 directory, 4 files</code></pre> Now we can call our application with mixed async and sync code and observe what is printed out. I used httpie</a> to make the requests. I've opened two consoles and made distinct HTTP requests to our application within the 20 seconds timeframe. This is the output: ❯ uvicorn app.application:app --workers 1 --log-level error 2021-07-07 20:08:57,962 9631c187-8f46-402a-b8ea-a15496643b81: started processing 2021-07-07 20:09:17,978 9631c187-8f46-402a-b8ea-a15496643b81 1: <Response [200]> 2021-07-07 20:09:37,987 9631c187-8f46-402a-b8ea-a15496643b81 2: <Response [200]> 2021-07-07 20:09:37,987 9631c187-8f46-402a-b8ea-a15496643b81: finished! 2021-07-07 20:09:37,988 694ee4be-a15a-49f6-ad60-7c140135a1f6: started processing 2021-07-07 20:09:57,997 694ee4be-a15a-49f6-ad60-7c140135a1f6 1: <Response [200]> 2021-07-07 20:10:18,004 694ee4be-a15a-49f6-ad60-7c140135a1f6 2: <Response [200]> 2021-07-07 20:10:18,004 694ee4be-a15a-49f6-ad60-7c140135a1f6: finished!</code></pre> As we can observe in the output that even though I've made both requests "in parallel" (same second) the server only accepted the request/started processing the second request (694ee4be-a15a-49f6-ad60-7c140135a1f6</code>) after the full execution of the first request (9631c187-8f46-402a-b8ea-a15496643b81</code>) which took a full 40 seconds. During the whole 40 seconds, there was no task switching and the worker event loop was completely blocked. All requests to the API are stale for the full 40 seconds, including requests to any other endpoints that might exist in other parts of the application. Even if the other requests don't call the external API, they cannot execute because the worker event loop is blocked. If we call the GET /healthcheck</code> endpoint it will not execute either. </div> Blocking Calls Break Async Using synchronous libraries like requests in async endpoints blocks the entire event loop. Even increasing workers won't solve this - your API becomes vulnerable to accidental DoS. Always use async-compatible libraries (like httpx) in async code. </div> </section> The way to solve this problem is to not let our workers get blocked. Our API should be fully async. For that, we need to replace the requests library with a library that supports async. Let's now implement a “v2” version of our example API, still calling the same fake external API that takes 20 seconds to process. Furthermore, we will again run Uvicorn with a single worker. Here is the code with the updated implementation: # # Async end-to-end # async def v2_internal_signing_op(op_num: int, request_id: str) -> None: """Calls external API endpoint and returns the response as a dict.""" async with httpx.AsyncClient() as session: response = await session.request("GET", EXTERNAL_API_ENDPOINT, timeout=2000) log.debug(f"{request_id} {op_num}: {response}") async def v2_sign_op1(request_id: str) -> None: await v2_internal_signing_op(1, request_id) async def v2_sign_op2(request_id: str) -> None: await v2_internal_signing_op(2, request_id) @app.get("/all-async") async def v2_root(): request_id = str(uuid.uuid4()) log.debug(f"{request_id}: started processing") await v2_sign_op1(request_id) await v2_sign_op2(request_id) log.debug(f"{request_id}: finished!") return {"message": "hello world"}</code></pre> Notice that I've replaced the requests library with the httpx library which supports async HTTP calls</a> and has an API that is very similar to the one requests provide. The code is functionally equivalent to our previous mixed implementation, but now we implemented async fully. Let's execute our API using the same commands as before. uvicorn app.application:app --workers 1</code></pre> Then calling the API using httpie, but to the fully async endpoint: http localhost:8000/all-async</code></pre> The console output is: 2021-07-07 23:30:21,673 da97310b-1d20-4082-8f90-b2e163523b83: started processing 2021-07-07 23:30:23,768 291f556e-038d-4230-8b3b-8e8270383e62: started processing 2021-07-07 23:30:41,718 da97310b-1d20-4082-8f90-b2e163523b83 1: <Response [200 OK]> 2021-07-07 23:30:43,781 291f556e-038d-4230-8b3b-8e8270383e62 1: <Response [200 OK]> 2021-07-07 23:31:01,740 da97310b-1d20-4082-8f90-b2e163523b83 2: <Response [200 OK]> 2021-07-07 23:31:01,740 da97310b-1d20-4082-8f90-b2e163523b83: finished! 2021-07-07 23:31:03,801 291f556e-038d-4230-8b3b-8e8270383e62 2: <Response [200 OK]> 2021-07-07 23:31:03,801 291f556e-038d-4230-8b3b-8e8270383e62: finished!</code></pre> We can observe in the output that both requests started processing immediately and they are still sequential in their own request lifecycle. The event loop of the Uvicorn worker is not blocked, that is why the second request could continue processing even though the external API did not finish its operation. Other requests, like the GET /healthcheck</code>, are not impacted by the slow execution of the external API. Overall our application continues to serve other requests independently on the external API. When using async Python one must be careful about what libraries to use. Even though a library might be very popular in the Python community, it doesn't mean that the library will play well in an async application. Choosing the right libraries will make the execution of the application more concurrent by not blocking the I/O loop. The overall throughput of the application will be better as more unrelated requests can be processed by the same Uvicorn worker. I've used async Python in some applications I maintain, and it was challenging to choose the right libraries to use. The team has to be on the watch for possible places in the code where the event loop may block. Even using the built-in Python logging library or a “print” statement is going to block the I/O loop. Usually, those blocking calls are negligible, but it is important to understand that they are there. I highly recommend also reading the official documentation</a> on other tips for developing async code in Python. Have you developed an async Python API, what was your experience?

Podcasts</h2>

The Pragmatic Engineer</a></strong> - Relevant content for sowftware engineers or anyone interested in working in tech.</li> </ul>

Topics I Write About</h2>
I explore content related to:</p>

General programming concepts</li>
Multimedia technology</li>
Open Source and FOSS initiatives</li>
Mechanical keyboards</li>
Technology in general</li> </ul>
Credits</h2>
This blog is powered by Zola</a>.</p>

Credits</h2>
This blog is powered by Zola</a>.</p>

Filter elements</h3>
Filter elements have a sink and a source pad always available.</p>
Filter elements generally have a single input (sink) pad and a single output (source) stream pad. The pads in a filter element are usually static (always available).</p>

Rafael Caricio

An Exercise in Agentic Coding: AV1 Encoder from Scratch in Rust

Learn by example: Making it easier to understand GStreamer

The Core
GStreamer is an object oriented framework,

My Late-Night Hack with iOS Shortcuts and AI

Readable Rust: Combinators vs. Match Clauses

Decompress content using Rust and flate2 without headaches

Blogroll

About Me

A brief introduction to GStreamer

Install GStreamer SRT plugin on macOS

Why not just use async Python for this API?

Rafael Caricio

An Exercise in Agentic Coding: AV1 Encoder from Scratch in Rust

Learn by example: Making it easier to understand GStreamer

The Core GStreamer is an object oriented framework,

My Late-Night Hack with iOS Shortcuts and AI

The problem</h2> As an immigrant in The Netherlands, I face two main challenges:</p>

The Solution: A 10-Minute Hack</h2> Building upon an existing shortcut I had for removing ads from web pages, I put together a multi-step workflow that does the following:</p>

Podcasts</h2> The Pragmatic Engineer</a></strong> - Relevant content for sowftware engineers or anyone interested in working in tech.</li> </ul>

Credits</h2> This blog is powered by Zola</a>.</p>

Filter elements</h3> Filter elements have a sink and a source pad always available.</p> Filter elements generally have a single input (sink) pad and a single output (source) stream pad. The pads in a filter element are usually static (always available).</p>

Install GStreamer SRT plugin on macOS

Why not just use async Python for this API?

The Core
GStreamer is an object oriented framework,

Podcasts</h2>

The Pragmatic Engineer</a></strong> - Relevant content for sowftware engineers or anyone interested in working in tech.</li> </ul>

Credits</h2>
This blog is powered by Zola</a>.</p>

Filter elements</h3>
Filter elements have a sink and a source pad always available.</p>
Filter elements generally have a single input (sink) pad and a single output (source) stream pad. The pads in a filter element are usually static (always available).</p>