投稿

Step-by-Step Guide: Interruption Management with OpenAI Realtime API

イメージ
Step-by-Step Guide: Interruption Management with OpenAI Realtime API This article introduces how to implement a conversation interruption feature using the OpenAI Realtime API. The details of the implementation are available in the GitHub repository . This implementation is based on the code from Azure-Samples/aoai-realtime-audio-sdk . A detailed explanation of the code can be found in this article . Audio Input In this implementation, we use the microphone and speaker of the local PC for audio input and output. The audio captured from the microphone is sent to the OpenAI Realtime API server for processing. To capture audio from the local PC’s microphone, we use the stream functionality of the pyaudio library. The following code sets up a stream for audio input: p = pyaudio . PyAudio ( ) input_default_input_index = p . get_default_input_device_info ( ) [ 'index' ] input_stream = p . open ( format = STREAM_FORMAT , channels ...

How to Convert PDFs to Markdown Using PyMuPDF4LLM and Its Evaluation

イメージ
How to Convert PDFs to Markdown Using PyMuPDF4LLM and Its Evaluation PyMuPDF4LLM is a library designed to convert PDFs into Markdown format. Here, I’ll share my experience testing this library. Installation Start by installing the library using the following command: pip install pymupdf4llm Usage The basic usage is quite simple, requiring just three lines of code to convert a PDF to Markdown: import pymupdf4llm md_text = pymupdf4llm . to_markdown ( "input.pdf" ) print ( md_text ) You can specify arguments to adjust how content is extracted. Extracting Text by Page By default, the entire PDF is converted into a single text output. However, you can extract text page by page by specifying page_chunks=True . md_text = pymupdf4llm . to_markdown ( "input.pdf" , page_chunks = True ) Extracting Images To extract images as files, use the write_images=True option: md_text = pymupdf4llm . to_markdown ( "input.pdf" , ...

Style your Kaggle Notebook

イメージ
Style your Kaggle Notebook I decided to restart Kaggle after two years, and to get myself pumped up, I started by designing the style for my notebook. If you’re getting tired of Markdown, give it a try! Styled Notebook Sample

OpenAI Realtime API: Points to keep in mind when executing functions

イメージ
Points to keep in mind when executing functions with a real-time API This time, I encountered a small issue when executing functions using the Realtime API, so I’m leaving a note here as a solution. The actual code is available in the GitHub repository below, so feel free to check it out for reference. Voice Chat Repository The above code is based on an Azure sample . I have provided an explanation of the Azure sample code here as well, so feel free to read it if you're interested. Flow of Function Execution 1. Preparing the Function First, specify the Tool to execute the function. In the following code, a function called get_your_info is defined. TOOLS = [ { "type" : "function" , "name" : "get_your_info" , "description" : "Get the information of your own.(e.g. name, age, hobby, favorite food, favorite color, favorite music, favorite movie, favorite game, favo...

OpnAI Realtime API: Conversing via Local Microphone and Speaker

イメージ
Conversing via Local Microphone and Speaker using Realtime API Several code samples using the Realtime API provided by OpenAI and Azure are available online. However, Python code is only available on Azure’s GitHub, and it assumes the use of an audio file as input. Therefore, I modified the code to accept real-time audio input from the local microphone using Python. The modified version is available on GitHub . Since the code is simple and concise, it should be easy to integrate into other projects. The original code is based on low_level_sample.py , and a detailed explanation is available in this article , which you can refer to. About the Modifications This article explains how to modify a Python application that processes audio to accept input from the local microphone and output audio data returned by the Realtime API through the local speaker. The implementation mainly uses the pyaudio library. The modifications consist of the following two points: ...

Prompt Caching: Comparing OpenAI, Anthropic, and Gemini

イメージ
Comparing Prompt Caching: OpenAI, Anthropic, and Gemini In recent years, the rapid development of large language models (LLMs) has led to significant increases in context window sizes. A context window refers to the amount of information a model can process at one time, and innovations like Retrieval-Augmented Generation (RAG), video, and image inputs have expanded the usable context length in LLMs. This evolution is aimed at handling more complex tasks and a wider range of information. In response, major providers have introduced “prompt caching” for efficient prompt management. Prompt caching stores previously used prompts and their results for reuse, avoiding repeated processing of the same tasks. This leads to faster processing times and cost savings. In this article, we will compare the prompt caching features of the key LLM providers: OpenAI, Anthropic, and Gemini, focusing on their specifications and differences. Models Supporting Prompt Caching Promp...

OpenAI Realtime API Python Code: Understanding the Low-Level Sample Code for Azure's Realtime Audio Python Code

イメージ
OpenAI Realtime API Python Code: Understanding the Low-Level Sample Code for Azure's Realtime Audio Python Code Introduction The “gpt-4o-realtime-preview” has been released. In addition to text and audio input/output, it also allows custom function calling via function calling. As of October 2, 2024, there are issues such as 403 errors, and it seems the API is not usable. This article will be updated once it becomes available. OpenAI has provided a JavaScript code sample on its website. Additionally, Azure has also published a Python code sample on GitHub. In this article, we will analyze Azure’s sample code, “ low_level_sample.py ,” to understand how it works. Libraries The required libraries are as follows: python-dotenv soundfile numpy scipy Code Explanation main Function In the main function, it first loads the dotenv file to retrieve the API key and endpoint: load_dotenv ( ) Next, it checks the arguments. This file is executed...