From af93960c4a808c291afa0efef5b148ab27418418 Mon Sep 17 00:00:00 2001 From: Christopher Fox Date: Tue, 4 Aug 2026 16:24:47 -0400 Subject: [PATCH 1/6] feat(stock-watchlist-agent-js): accept image inputs and annotate them on spans Adds an image input path to the JS stock watchlist demo. Arguments that look like images (by extension, http(s) URL, or data: URL) are translated to ticker symbols by an LLM before research begins, and the image bytes are attached to the resulting llm span via imageParts. - vision.js: identify_ticker is an llm-kind span annotated with imageParts: [{mimeType, content}]; images are always materialized as base64 (URLs downloaded) so the same bytes go to OpenAI and onto the span - orchestrator: image resolution runs inside the root analyze_portfolio span so vision and research share one trace - main.js: mixed ticker/image args, --image override, resolution summary - logos/: three small wordmark images so the image path is runnable as-is - default model for all call sites is now gpt-5.4-nano Co-Authored-By: Claude Opus 5 --- .../stock-watchlist-agent-js/.env.example | 8 +- test-apps/stock-watchlist-agent-js/README.md | 26 +++- .../stock-watchlist-agent-js/logos/apple.png | Bin 0 -> 746 bytes .../stock-watchlist-agent-js/logos/google.png | Bin 0 -> 923 bytes .../stock-watchlist-agent-js/logos/nvidia.png | Bin 0 -> 897 bytes .../src/agents/orchestrator.js | 24 ++- .../src/agents/responses-agent.js | 2 +- .../src/agents/searcher.js | 2 +- .../src/agents/vision.js | 140 ++++++++++++++++++ .../stock-watchlist-agent-js/src/evals.js | 4 +- .../stock-watchlist-agent-js/src/main.js | 75 ++++++++-- .../stock-watchlist-agent-js/src/models.js | 27 ++++ 12 files changed, 285 insertions(+), 23 deletions(-) create mode 100644 test-apps/stock-watchlist-agent-js/logos/apple.png create mode 100644 test-apps/stock-watchlist-agent-js/logos/google.png create mode 100644 test-apps/stock-watchlist-agent-js/logos/nvidia.png create mode 100644 test-apps/stock-watchlist-agent-js/src/agents/vision.js diff --git a/test-apps/stock-watchlist-agent-js/.env.example b/test-apps/stock-watchlist-agent-js/.env.example index 9a5a744f..42b6479e 100644 --- a/test-apps/stock-watchlist-agent-js/.env.example +++ b/test-apps/stock-watchlist-agent-js/.env.example @@ -1,9 +1,11 @@ # OpenAI OPENAI_API_KEY=sk-... # Optional model overrides -# OPENAI_MODEL=gpt-4o -# OPENAI_SEARCH_MODEL=gpt-4o-mini -# OPENAI_EVAL_MODEL=gpt-4o-mini +# OPENAI_MODEL=gpt-5.4-nano +# OPENAI_SEARCH_MODEL=gpt-5.4-nano +# OPENAI_EVAL_MODEL=gpt-5.4-nano +# Model used to translate image inputs into ticker symbols +# OPENAI_VISION_MODEL=gpt-5.4-nano # Datadog LLM Observability DD_API_KEY= diff --git a/test-apps/stock-watchlist-agent-js/README.md b/test-apps/stock-watchlist-agent-js/README.md index 8c54476a..a4fe8202 100644 --- a/test-apps/stock-watchlist-agent-js/README.md +++ b/test-apps/stock-watchlist-agent-js/README.md @@ -6,6 +6,8 @@ JavaScript translation of `test-apps/stock-watchlist-agent`, built with OpenAI's ``` llmobs.trace(kind="agent", name="analyze_portfolio") ← evals attach here +├── resolve_tickers_from_images (workflow) ← only when image inputs are given +│ └── identify_ticker (llm, one per image) → OpenAI vision call, image_parts annotated └── orchestrator (OpenAI Responses ReAct loop) ├── delegate_research (tool, batched tickers) │ └── stock_researcher (OpenAI Responses ReAct loop) @@ -17,6 +19,8 @@ llmobs.trace(kind="agent", name="analyze_portfolio") ← evals attach her └── stock_researcher ... ``` +Inputs can be ticker symbols, images, or a mix of both. When images are provided, a vision step runs first inside the root `analyze_portfolio` span: one LLM call per image identifies the public company shown and returns its ticker symbol, and the resolved tickers are merged with any tickers passed directly. Images that cannot be matched to a public company come back as `UNKNOWN` and are skipped. Image resolution and research therefore share a single trace. + The orchestrator plans how to batch tickers by sector/theme, delegates batches to researcher agents, and synthesizes a portfolio briefing. Each researcher runs a multi-step ReAct loop with four research tools backed by OpenAI web search. Post-run evaluations (completeness, sentiment consistency, factual grounding) are submitted to LLM Observability. ## Development @@ -42,9 +46,27 @@ npm install /path/to/dd-trace-js/packages/dd-trace ```bash # Either export OPENAI_API_KEY or set it in .env + +# a) All three inputs are ticker symbols npm start -- AAPL GOOGL NVDA + +# b) First two inputs are images, third is a ticker symbol +npm start -- logos/apple.png logos/google.png NVDA + +# Image URLs work too, and --image forces an argument to be read as an image +npm start -- https://example.com/nvidia-logo.jpg --image logos/apple.png ``` +The `logos/` directory holds small sample wordmark images for Apple, Google, and NVIDIA so the image path can be run without supplying your own files. + +Arguments ending in `.png`, `.jpg`, `.jpeg`, `.gif`, or `.webp`, plus any `http(s)://` or `data:image/...` argument, are treated as images automatically. Use `--image ` to force an argument to be read as an image. Every image is read into memory as base64 (URLs are downloaded first) so the same bytes can be sent to OpenAI and attached to the trace. Set `OPENAI_VISION_MODEL` to override the model used for image-to-ticker translation (defaults to `OPENAI_MODEL`, then `gpt-5.4-nano`). + +### Images on spans + +`identify_ticker` is annotated as an `llm`-kind span whose user message carries `imageParts: [{ mimeType, content }]`, which the SDK emits as `image_parts: [{ mime_type, content }]`. + +This only works with an SDK version that supports image parts, and it must be done by hand: provider auto-instrumentation does not capture images, and `imageParts` is ignored on non-`llm` span kinds (`task`, `workflow`, `agent`), where input is tagged as a plain value. + ## Running with Datadog LLM Observability ```bash @@ -102,6 +124,7 @@ When LLMObs is enabled, three evaluations run after each analysis and are submit ## Project Structure ``` +logos/ # Small sample images for the image-input path src/ ├── main.js # CLI entry point, eval runner ├── observability.js # .env loading, dd-trace-js initialization, LLMObs helpers @@ -111,5 +134,6 @@ src/ ├── orchestrator.js # ReAct orchestrator, delegation tool, agent span ├── researcher.js # Per-batch research agent with 4 tools ├── responses-agent.js # Generic Responses API function-calling loop - └── searcher.js # OpenAI Responses API web search helper + ├── searcher.js # OpenAI Responses API web search helper + └── vision.js # Image → ticker symbol translation ``` diff --git a/test-apps/stock-watchlist-agent-js/logos/apple.png b/test-apps/stock-watchlist-agent-js/logos/apple.png new file mode 100644 index 0000000000000000000000000000000000000000..d06eb69f1442667c9137567cc7bdf5b0678defe9 GIT binary patch literal 746 zcmVI=&@M`e|0825m$)+YXe@id z4g2|pwqTEr&5CacOLeS7wi;(AD858nx_;+~KH7QMb%LJfadv_`?(+}zj_lgq?rRWy zu&^Nu_h=TuvqUHB7Y}H7O;u1vm(L5S7aSNfb!eyLXcj^KEANcCtUbn?MBI-KWr-7f z=Dur3Sp*w0QF%(dQ58Jk)x>LvpuIgycGyR=2=X;NxV65U4O*OWlYYgugr#n}^G!F( zBFG&{<$|D>VAEO&LWi^*YKW3f{>DZA<>_4yKxd z`LD`S+;m%7n1iio4sz4=PYP3opKVFgR<>yI?s$ur)djgD#}ej=zKGykH}35yi{RSC zhrfTod{}m8WbU(6nwWae?<9E^!8zGUreEII-0J)|%6iBxvHS1ih1)?k&mzc2@9ge{ zZK6-Q9chIfh12P$nNX-L)t3rclJCh3+v06_>s5P|>z0000000000 c0Q`^q0*`aR1-5I8f&c&j07*qoM6N<$f+enBaR2}S literal 0 HcmV?d00001 diff --git a/test-apps/stock-watchlist-agent-js/logos/google.png b/test-apps/stock-watchlist-agent-js/logos/google.png new file mode 100644 index 0000000000000000000000000000000000000000..c26905c468bc3d2eb80571864419f4b00e43c713 GIT binary patch literal 923 zcmV;M17!S(P)x9RlezhUuJ~@D_gaF|9LL9Y(*1MsRzZg)#X1-)9#V)(jMAWRp<~c);iU-l5FN5G4_@q` zhtSA_78yu~twX^^FhP)!Y#xg0pvL8wUD)4$ha|h5h<_ zzkhEvjUWhuAP9mW2!bF8f*}4O-S-Mh`Kz1$xDN0uP=BY2z?coI!QH0i530lAG`M3L zY*xc*a1RIuLTPXaolP8XzKo9#VKk^?20mjaJ+|pH4%T24U_He|44@oFgGT^XED>1* z=wzaOYWdxW{jQ`V*9-XzI{}^ugYA!(@)wmG><5_P>h@$`BR&q8Zc~)?O;n{5lEDW2 zkD#1YZ4jG`=R5SARnV0IpN?8`-Zf~qr`1Nq1`YdQ9S++q=eJJ}aj&ky=+=7EwlSgN zkLdHZo+4D!9v#8Mo^84UuE7kzo3?WR9~I9LyLS0uj%!#2B`oUm0QWt*0bs3B(+$^P z4$7Tc#Mt!w*(MR6L2Nc(dB&}HbOPWtFDcug&QCrA<)rEeR%C+**&p;>#K_3{DA#+1 z!Fqt7_yT?=tvUxGDH+VzS#Q94EgcgEF|Ob7g@p1d$?*_42K()ZOykMRI>U8T7>tP( z3VF8)0zf1L@@i82#%{Y*eWK!Ww2M|UCE0HJwwVea zB^jxTb*YLieT?f?VGtYRk3bBv5%GzM4wHF^^U+}SK3OODI=i6{2DDK>>T14ZTIp0T zrpJs&+mmF`og@aEB--HJKGoZTo9(P1>vGoo#R;AEXtpihX}TfJVhrYdTes;XND#L= z_>93gZEIFJvL13i!XEyI?4)0+)1Y1FH9u0P!SV1 zn2-`zK6vsfzFUimr xaLee+<%P+VMi2x+5ClOG1VIo4K@k5he*jFaQc{?bH^~41002ovPDHLkV1m^RyK4Xd literal 0 HcmV?d00001 diff --git a/test-apps/stock-watchlist-agent-js/logos/nvidia.png b/test-apps/stock-watchlist-agent-js/logos/nvidia.png new file mode 100644 index 0000000000000000000000000000000000000000..fa43bcccd26e541ba235e4858844ba2cdb300ed5 GIT binary patch literal 897 zcmV-{1AhF8P)axn@oVeS3uhwd$&{Uw$PoU0DlgKxFyb5=@0d~0ngZZ>%0009F zNkli`S05AfPlfO7xs^MK#rF-kd+SA_!?9#{of+2IzzWg3WC z4Klz8hbCcw>$U(9bKjOiO>#i^o&jRD;YS!Cyb++80vz%1F}=tE`7IB8h6A^OfR`Q^ z1AOG+gi4tQen|uIE4bPh4_tX*7vO^a*a5gR9(Zd?Po7Wh=LA?v;H#9oa3CBAu;JRc z1~5+pF^}!%1o!~pZICWu;Hdy3uADIs0nTzjGh&Yb@vFQ@LUaiO7XmCNwvDj^PV&G# zfY}Dv#|bzN(Ip5R!Mf*wR3@5NLx4dZSm)a>z#*184$&nDe8mfm1s>Wqp2GT39*DPp zxdA4q?S|+Q1irk1D*@sTa&8Ei7c?f>x=`| zsnSu1y@Y@rT0kt&mZ@n=vjy0pyIa2DLKhE=Xi>31oLe~Y`&N+!mT~9sz?i#z9H2`G zSeoKOEzwh{;E5&~lHu6quME8u%~`7OYT-Eq1Bx`ct)jwcR?)9)PT z*1~M<;+j7|%z^+bxX?HtHt;-h2b_PegDnqS1_9#PX*IS$^2qv@1=dnKN$i2Npa%fr zs*zYAZ6bSa?q`9m)LuN!o??p!(gTJA#=JSV{{+UH(+o}tpGgoPE;It{Zmb#LPQumV zu__-I?XheY1W4C)ss;h)d0^@3J6vVt0AY`dwxyO~fDI4iyM2}cO0HyVpO1^SrHbwn z2dpFlqz7_$`8?2XeT=j zWg*%;5%=AiUOX`Jz?Pr*ncrqU_54^$(P%UpjYgx { const now = formatUtcNow() - const prompt = `Analyze these stock tickers: ${tickers.join(', ')}. Current time: ${now}` const spanContext = exportSpan(span) - annotate(span, { inputData: tickers, metadata: { generated_at: now } }) + let tickers = inputTickers + let identifications = [] + if (images.length > 0) { + const resolved = await resolveTickersFromImages(images) + identifications = resolved.identifications + tickers = [...new Set([...inputTickers, ...resolved.tickers])] + if (onImagesResolved) onImagesResolved(identifications) + if (tickers.length === 0) { + throw new Error('No ticker symbols could be identified from the provided image(s)') + } + } + + const prompt = `Analyze these stock tickers: ${tickers.join(', ')}. Current time: ${now}` + annotate(span, { + inputData: images.length > 0 ? { tickers: inputTickers, images } : inputTickers, + metadata: { generated_at: now, resolved_tickers: tickers }, + }) const briefing = await traceSpan({ kind: 'agent', name: 'orchestrator' }, async orchestratorSpan => { annotate(orchestratorSpan, { inputData: prompt, metadata: { generated_at: now } }) const result = await runResponsesAgent({ @@ -128,7 +144,7 @@ async function analyzePortfolio (tickers) { return validated }) annotate(span, { outputData: briefing }) - return { briefing, spanContext } + return { briefing, spanContext, tickers, identifications } }) } diff --git a/test-apps/stock-watchlist-agent-js/src/agents/responses-agent.js b/test-apps/stock-watchlist-agent-js/src/agents/responses-agent.js index dbdfdea8..fe15c8af 100644 --- a/test-apps/stock-watchlist-agent-js/src/agents/responses-agent.js +++ b/test-apps/stock-watchlist-agent-js/src/agents/responses-agent.js @@ -4,7 +4,7 @@ const { annotate, traceSpan } = require('../observability') const OpenAI = require('openai') const client = new OpenAI() -const DEFAULT_MODEL = process.env.OPENAI_MODEL || 'gpt-4o' +const DEFAULT_MODEL = process.env.OPENAI_MODEL || 'gpt-5.4-nano' function jsonSchemaFormat (name, schema) { return { diff --git a/test-apps/stock-watchlist-agent-js/src/agents/searcher.js b/test-apps/stock-watchlist-agent-js/src/agents/searcher.js index ab9372b5..9f783e86 100644 --- a/test-apps/stock-watchlist-agent-js/src/agents/searcher.js +++ b/test-apps/stock-watchlist-agent-js/src/agents/searcher.js @@ -13,7 +13,7 @@ const SEARCH_INSTRUCTIONS = [ async function search (query) { const response = await client.responses.create({ - model: process.env.OPENAI_SEARCH_MODEL || 'gpt-4o-mini', + model: process.env.OPENAI_SEARCH_MODEL || 'gpt-5.4-nano', instructions: SEARCH_INSTRUCTIONS, input: query, tools: [{ type: 'web_search' }], diff --git a/test-apps/stock-watchlist-agent-js/src/agents/vision.js b/test-apps/stock-watchlist-agent-js/src/agents/vision.js new file mode 100644 index 00000000..7d66b50b --- /dev/null +++ b/test-apps/stock-watchlist-agent-js/src/agents/vision.js @@ -0,0 +1,140 @@ +'use strict' + +const fs = require('node:fs') +const path = require('node:path') +const { annotate, traceSpan } = require('../observability') +const { tickerFromImageSchema, validateTickerFromImage } = require('../models') +const { jsonSchemaFormat } = require('./responses-agent') +const OpenAI = require('openai') + +const client = new OpenAI() +const DEFAULT_VISION_MODEL = process.env.OPENAI_VISION_MODEL || process.env.OPENAI_MODEL || 'gpt-5.4-nano' + +const IMAGE_MIME_TYPES = { + '.png': 'image/png', + '.jpg': 'image/jpeg', + '.jpeg': 'image/jpeg', + '.gif': 'image/gif', + '.webp': 'image/webp', +} + +const VISION_PROMPT = `\ + +You identify the publicly traded company shown in an image and return its stock ticker symbol. + + + +1. Read every signal in the image: logos, wordmarks, product names, storefronts, chart labels, ticker tape text +2. Decide which single public company the image refers to +3. Map that company to the ticker symbol of its primary US listing (use the local exchange symbol if it is not US-listed) +4. Report how confident you are and what evidence you used + + + +- Return the ticker in uppercase with no exchange prefix (AAPL, not NASDAQ:AAPL) +- Prefer the parent company's ticker when the image shows a brand or product (e.g. Instagram -> META) +- If the image shows no identifiable public company, set ticker to "UNKNOWN" and confidence to "low" + + + +Return JSON only matching the TickerFromImage schema. +` + +function isHttpUrl (value) { + return /^https?:\/\//i.test(value) +} + +function looksLikeImage (value) { + if (isHttpUrl(value) || value.startsWith('data:image/')) return true + return Object.keys(IMAGE_MIME_TYPES).includes(path.extname(value).toLowerCase()) +} + +const IMAGE_QUESTION = 'Which publicly traded company does this image show? Return its ticker symbol.' + +async function loadImage (imageInput) { + if (imageInput.startsWith('data:image/')) { + const match = imageInput.match(/^data:(image\/[\w+.-]+);base64,(.*)$/) + if (!match) { + throw new Error(`Malformed image data URL: ${imageInput.slice(0, 32)}...`) + } + return { mimeType: match[1], base64: match[2], source: 'data-url' } + } + + if (isHttpUrl(imageInput)) { + const response = await fetch(imageInput) + if (!response.ok) { + throw new Error(`Failed to download image ${imageInput}: HTTP ${response.status}`) + } + const mimeType = (response.headers.get('content-type') || 'image/png').split(';')[0] + const buffer = Buffer.from(await response.arrayBuffer()) + return { mimeType, base64: buffer.toString('base64'), source: 'url' } + } + + const filePath = path.resolve(imageInput) + const extension = path.extname(filePath).toLowerCase() + const mimeType = IMAGE_MIME_TYPES[extension] + if (!mimeType) { + throw new Error(`Unsupported image type "${extension}" for ${imageInput}`) + } + if (!fs.existsSync(filePath)) { + throw new Error(`Image not found: ${imageInput}`) + } + return { mimeType, base64: fs.readFileSync(filePath).toString('base64'), source: 'file' } +} + +async function identifyTicker (imageInput, model = DEFAULT_VISION_MODEL) { + // Annotated as an `llm` span with imageParts: the LLMObs SDK only renders images + // on manually annotated llm-kind messages, not from provider auto-instrumentation. + return traceSpan({ kind: 'llm', name: 'identify_ticker', modelName: model, modelProvider: 'openai' }, async span => { + const { mimeType, base64, source } = await loadImage(imageInput) + annotate(span, { + inputData: [ + { role: 'system', content: VISION_PROMPT }, + { + role: 'user', + content: IMAGE_QUESTION, + imageParts: [{ mimeType, content: base64 }], + }, + ], + metadata: { model, image_source: source, image_input: imageInput }, + }) + + const response = await client.responses.create({ + model, + instructions: VISION_PROMPT, + input: [ + { + role: 'user', + content: [ + { type: 'input_text', text: IMAGE_QUESTION }, + { type: 'input_image', image_url: `data:${mimeType};base64,${base64}`, detail: 'auto' }, + ], + }, + ], + text: { format: jsonSchemaFormat('TickerFromImage', tickerFromImageSchema) }, + }) + + const result = validateTickerFromImage(JSON.parse((response.output_text || '').trim())) + const identified = { ...result, ticker: result.ticker.toUpperCase(), source: imageInput } + annotate(span, { outputData: [{ role: 'assistant', content: JSON.stringify(identified) }] }) + return identified + }) +} + +async function resolveTickersFromImages (imageInputs) { + return traceSpan({ kind: 'workflow', name: 'resolve_tickers_from_images' }, async span => { + annotate(span, { inputData: imageInputs }) + const identifications = await Promise.all(imageInputs.map(image => identifyTicker(image))) + const recognized = identifications.filter(item => item.ticker !== 'UNKNOWN') + const tickers = [...new Set(recognized.map(item => item.ticker))] + annotate(span, { outputData: { tickers, identifications } }) + return { tickers, identifications } + }) +} + +module.exports = { + identifyTicker, + resolveTickersFromImages, + looksLikeImage, + VISION_PROMPT, +} diff --git a/test-apps/stock-watchlist-agent-js/src/evals.js b/test-apps/stock-watchlist-agent-js/src/evals.js index 780217ec..39a464f9 100644 --- a/test-apps/stock-watchlist-agent-js/src/evals.js +++ b/test-apps/stock-watchlist-agent-js/src/evals.js @@ -51,7 +51,7 @@ async function runBooleanJudge ({ name, prompt, outputData }) { return traceSpan({ kind: 'task', name }, async span => { annotate(span, { inputData: { output_data: outputData } }) const response = await client.responses.create({ - model: process.env.OPENAI_EVAL_MODEL || 'gpt-4o-mini', + model: process.env.OPENAI_EVAL_MODEL || 'gpt-5.4-nano', input: prompt.replace('{{output_data}}', JSON.stringify(outputData, null, 2)), text: { format: jsonSchemaFormat(name, booleanJudgeSchema) }, }) @@ -69,7 +69,7 @@ async function runScoreJudge ({ name, prompt, outputData, minThreshold }) { return traceSpan({ kind: 'task', name }, async span => { annotate(span, { inputData: { output_data: outputData } }) const response = await client.responses.create({ - model: process.env.OPENAI_EVAL_MODEL || 'gpt-4o-mini', + model: process.env.OPENAI_EVAL_MODEL || 'gpt-5.4-nano', input: prompt.replace('{{output_data}}', JSON.stringify(outputData, null, 2)), text: { format: jsonSchemaFormat(name, scoreJudgeSchema) }, }) diff --git a/test-apps/stock-watchlist-agent-js/src/main.js b/test-apps/stock-watchlist-agent-js/src/main.js index 6847db8f..0e321f0b 100755 --- a/test-apps/stock-watchlist-agent-js/src/main.js +++ b/test-apps/stock-watchlist-agent-js/src/main.js @@ -3,8 +3,22 @@ const { flush, isLLMObsEnabled, mlApp, site, env } = require('./observability') const { analyzePortfolio } = require('./agents/orchestrator') +const { looksLikeImage } = require('./agents/vision') const { runEvaluations } = require('./evals') +const USAGE = [ + 'Usage: node src/main.js [TICKER|IMAGE ...]', + '', + 'Inputs may be ticker symbols, image files, or image URLs (mixed freely).', + 'Images are translated to ticker symbols by an LLM before research begins.', + 'Use --image to force an argument to be treated as an image.', + '', + 'Examples:', + ' node src/main.js AAPL GOOGL NVDA', + ' node src/main.js ./logos/apple.png https://example.com/nvidia-logo.jpg', + ' node src/main.js AAPL --image ./logos/tesla.png', +].join('\n') + function printBriefing (briefing) { console.log('\n' + '='.repeat(60)) console.log(' STOCK WATCHLIST BRIEFING') @@ -44,30 +58,68 @@ function printBriefing (briefing) { console.log('\n' + '='.repeat(60) + '\n') } +function printIdentifications (identifications) { + console.log('\nImage inputs resolved to tickers:') + for (const item of identifications) { + const label = item.ticker === 'UNKNOWN' ? 'UNKNOWN (skipped)' : item.ticker + console.log(` ${item.source} -> ${label} (${item.company_name}, confidence: ${item.confidence})`) + console.log(` ${item.evidence}`) + } +} + function parseArgs (argv) { const args = argv.slice(2) if (args.includes('-h') || args.includes('--help')) { - console.log('Usage: node src/main.js [TICKER ...]') - console.log('Example: node src/main.js AAPL GOOGL NVDA') + console.log(USAGE) process.exit(0) } - return args.map(ticker => ticker.toUpperCase()) + + const tickers = [] + const images = [] + for (let i = 0; i < args.length; i++) { + const arg = args[i] + if (arg === '--image') { + const value = args[++i] + if (!value) { + throw new Error('--image requires a file path or URL') + } + images.push(value) + } else if (looksLikeImage(arg)) { + images.push(arg) + } else { + tickers.push(arg.toUpperCase()) + } + } + return { tickers, images } } -async function main (tickers) { +async function main (inputTickers, images = []) { if (!process.env.OPENAI_API_KEY) { throw new Error('OPENAI_API_KEY is required') } - console.log(`Analyzing ${tickers.length} ticker(s): ${tickers.join(', ')}`) + const described = [ + ...inputTickers, + ...images.map(image => `${image} (image)`), + ] + console.log(`Analyzing ${described.length} input(s): ${described.join(', ')}`) if (isLLMObsEnabled()) { console.log(`LLMObs enabled: ml_app=${mlApp}, site=${site}, env=${env}`) } else { console.log('LLMObs disabled: set DD_API_KEY (or DD_LLMOBS_ENABLED=true) to submit traces') } + if (images.length > 0) { + console.log(`Translating ${images.length} image input(s) to ticker symbols...`) + } console.log('Running parallel analysis with web search...\n') - const { briefing, spanContext } = await analyzePortfolio(tickers) + const { briefing, spanContext, tickers } = await analyzePortfolio(inputTickers, { + images, + onImagesResolved: identifications => { + printIdentifications(identifications) + console.log('') + }, + }) if (isLLMObsEnabled() && spanContext) { console.log(`LLMObs trace context: trace_id=${spanContext.traceId}, span_id=${spanContext.spanId}`) } @@ -81,16 +133,16 @@ async function main (tickers) { } async function cli () { - const tickers = parseArgs(process.argv) - if (tickers.length === 0) { - console.error('Error: provide at least one ticker symbol') - console.error('Usage: node src/main.js [TICKER ...]') + const { tickers, images } = parseArgs(process.argv) + if (tickers.length === 0 && images.length === 0) { + console.error('Error: provide at least one ticker symbol or image') + console.error(USAGE) process.exitCode = 1 return } try { - await main(tickers) + await main(tickers, images) } finally { flush() } @@ -105,5 +157,6 @@ if (require.main === module) { module.exports = { main, + parseArgs, printBriefing, } diff --git a/test-apps/stock-watchlist-agent-js/src/models.js b/test-apps/stock-watchlist-agent-js/src/models.js index c40c28aa..a411ea13 100644 --- a/test-apps/stock-watchlist-agent-js/src/models.js +++ b/test-apps/stock-watchlist-agent-js/src/models.js @@ -48,6 +48,18 @@ const portfolioBriefingSchema = { required: ['analyses', 'market_overview', 'highlights', 'generated_at'], } +const tickerFromImageSchema = { + type: 'object', + additionalProperties: false, + properties: { + ticker: { type: 'string' }, + company_name: { type: 'string' }, + confidence: { type: 'string', enum: ['high', 'medium', 'low'] }, + evidence: { type: 'string' }, + }, + required: ['ticker', 'company_name', 'confidence', 'evidence'], +} + function assertString (value, path) { if (typeof value !== 'string' || value.length === 0) { throw new Error(`${path} must be a non-empty string`) @@ -98,10 +110,25 @@ function validatePortfolioBriefing (briefing) { return briefing } +function validateTickerFromImage (result) { + if (!result || typeof result !== 'object' || Array.isArray(result)) { + throw new Error('TickerFromImage must be an object') + } + assertString(result.ticker, 'ticker') + assertString(result.company_name, 'company_name') + if (!['high', 'medium', 'low'].includes(result.confidence)) { + throw new Error('confidence must be high, medium, or low') + } + assertString(result.evidence, 'evidence') + return result +} + module.exports = { stockAnalysisSchema, researchBatchResultSchema, portfolioBriefingSchema, + tickerFromImageSchema, validateResearchBatchResult, validatePortfolioBriefing, + validateTickerFromImage, } From 904dc378a8edc50c79c31f360d4d77658614aefe Mon Sep 17 00:00:00 2001 From: Christopher Fox Date: Fri, 7 Aug 2026 14:10:46 -0400 Subject: [PATCH 2/6] chore(stock-watchlist-agent-js): require dd-trace ^6.10.0 for image parts Image parts on LLMObs spans shipped in dd-trace 6.10.0. The previous lockfile pinned 6.4.0, so a fresh install resolved a version that silently ignores imageParts and rendered no image on the span. Co-Authored-By: Claude Opus 5 --- .../package-lock.json | 63 +++++++++---------- .../stock-watchlist-agent-js/package.json | 2 +- 2 files changed, 32 insertions(+), 33 deletions(-) diff --git a/test-apps/stock-watchlist-agent-js/package-lock.json b/test-apps/stock-watchlist-agent-js/package-lock.json index 6fc0e734..d2f6c4d9 100644 --- a/test-apps/stock-watchlist-agent-js/package-lock.json +++ b/test-apps/stock-watchlist-agent-js/package-lock.json @@ -8,7 +8,7 @@ "name": "stock-watchlist-agent-js", "version": "0.1.0", "dependencies": { - "dd-trace": "latest", + "dd-trace": "^6.10.0", "dotenv": "^16.4.7", "openai": "^4.104.0" }, @@ -20,9 +20,9 @@ } }, "node_modules/@datadog/flagging-core": { - "version": "1.2.1", - "resolved": "https://registry.npmjs.org/@datadog/flagging-core/-/flagging-core-1.2.1.tgz", - "integrity": "sha512-qeDkki9fFlqyoZBrn7tneT6pZ04EKKvf3xxisYw1a74zbJihvQui/ARUsjXCurRpzpFqGGTJw/oz+HnXaKhcdw==", + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/@datadog/flagging-core/-/flagging-core-2.0.2.tgz", + "integrity": "sha512-2+oWyqz/EMNXtsgyW3NtueFgL0TciWInyMg/6bEUZhfr7UgPzCQ88Nag9FGoPCig19F/tHXE5hLiQccjkRUMWQ==", "license": "Apache-2.0", "optional": true, "dependencies": { @@ -77,13 +77,13 @@ } }, "node_modules/@datadog/openfeature-node-server": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/@datadog/openfeature-node-server/-/openfeature-node-server-2.0.0.tgz", - "integrity": "sha512-Ummu/Bd7ZJpCCNdFnZUt/JI+L1+8OMK53+MyIXbS7dCt4JXWWwelbpzSGqmC2jWbWQ/mTo4hEfnFw3kYOINbXA==", + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/@datadog/openfeature-node-server/-/openfeature-node-server-2.0.2.tgz", + "integrity": "sha512-647eJuiOVzCEk50His94wjSlOgJCbHIJv4cAW425eW3GqQJC2Y73qGBYV1/s9oBsAmsUNa90tc+ETunfUH2hKQ==", "license": "Apache-2.0", "optional": true, "dependencies": { - "@datadog/flagging-core": "1.2.1" + "@datadog/flagging-core": "2.0.2" }, "engines": { "node": ">=18.0.0" @@ -93,16 +93,15 @@ } }, "node_modules/@datadog/pprof": { - "version": "5.15.1", - "resolved": "https://registry.npmjs.org/@datadog/pprof/-/pprof-5.15.1.tgz", - "integrity": "sha512-4mI750tX6okNROS4YKvGQjyAQ+VqfzDqzysCytOJhL2E2ktK/q4M0PtC7j70tiirGb/fO9fjma3IMNSRLYX+xQ==", - "hasInstallScript": true, + "version": "5.17.0", + "resolved": "https://registry.npmjs.org/@datadog/pprof/-/pprof-5.17.0.tgz", + "integrity": "sha512-BuedZ4vHzmQkitMMdIhVLQ+nPpHl7WRwcuumJeXt0ylhwGGwt7Kf+4CoS1tDB0FljJXi4izweFfFFTirrijP5w==", "license": "Apache-2.0", "optional": true, "dependencies": { "node-gyp-build": "^4.8.4", "pprof-format": "^2.2.1", - "source-map": "^0.7.4" + "source-map": "^0.8.0" }, "engines": { "node": ">=16" @@ -198,17 +197,17 @@ } }, "node_modules/@openfeature/core": { - "version": "1.11.0", - "resolved": "https://registry.npmjs.org/@openfeature/core/-/core-1.11.0.tgz", - "integrity": "sha512-P0u3/ht/oZCQT89fOed+laLk0kZR529a825cS02uPDglxXbE97irWYpDAeRGGVETIzKfuy+H2g8c3Ccv/tXJNQ==", + "version": "1.12.0", + "resolved": "https://registry.npmjs.org/@openfeature/core/-/core-1.12.0.tgz", + "integrity": "sha512-7PCPzyd1OC19begz30+CRknFB0ChtyaZUhk7YWrk+Bov1fpVw0HYkTXtoxxsvPfDZB9P9WsT7uixN+Z8f9+bcA==", "license": "Apache-2.0", "optional": true, "peer": true }, "node_modules/@openfeature/server-sdk": { - "version": "1.22.0", - "resolved": "https://registry.npmjs.org/@openfeature/server-sdk/-/server-sdk-1.22.0.tgz", - "integrity": "sha512-YBrf6SQkn0FNB/dRAtLEs41dvFMUE8CrQTwI+iLaMFUIqWlqGNJfGnulKSneEKS+2OgKTAC6DdmKcZ6tK7kBcg==", + "version": "1.23.0", + "resolved": "https://registry.npmjs.org/@openfeature/server-sdk/-/server-sdk-1.23.0.tgz", + "integrity": "sha512-JWeLvltJIV0AFgOfbw7hK9b9Rw/5wWq+RMhCIU9sJtAqarxr8hSPglG+JAGGhYdElAz3Qwe9W3ApWyPNkQEyqg==", "license": "Apache-2.0", "optional": true, "peer": true, @@ -216,7 +215,7 @@ "node": ">=20" }, "peerDependencies": { - "@openfeature/core": "^1.11.0" + "@openfeature/core": "^1.12.0" } }, "node_modules/@opentelemetry/api": { @@ -681,13 +680,13 @@ } }, "node_modules/dd-trace": { - "version": "6.4.0", - "resolved": "https://registry.npmjs.org/dd-trace/-/dd-trace-6.4.0.tgz", - "integrity": "sha512-aRcf+OXejmPmOz5mTg5zkBTAONv+JwHeszdu/sxPEuZvkBYfjHod5XChgSV0rEDwOWvSbe0kaLPdWM/umIftsw==", + "version": "6.10.0", + "resolved": "https://registry.npmjs.org/dd-trace/-/dd-trace-6.10.0.tgz", + "integrity": "sha512-sDRIz4mlwi6v48UatXtfuhG5rU/ymqddVqleUhBwMmGglEoW+CWyNklgQwAKfaZt4agU1kS0apoR/8JIeY0MIw==", "license": "(Apache-2.0 OR BSD-3-Clause)", "dependencies": { "dc-polyfill": "^0.1.11", - "import-in-the-middle": "^3.3.1", + "import-in-the-middle": "^3.3.2", "opentracing": ">=0.14.7" }, "engines": { @@ -698,8 +697,8 @@ "@datadog/native-appsec": "11.0.1", "@datadog/native-iast-taint-tracking": "4.2.0", "@datadog/native-metrics": "3.1.2", - "@datadog/openfeature-node-server": "2.0.0", - "@datadog/pprof": "5.15.1", + "@datadog/openfeature-node-server": "2.0.2", + "@datadog/pprof": "5.17.0", "@datadog/wasm-js-rewriter": "5.0.1", "@opentelemetry/api": ">=1.0.0 <1.10.0", "@opentelemetry/api-logs": "<1.0.0", @@ -1168,16 +1167,16 @@ } }, "node_modules/pprof-format": { - "version": "2.2.2", - "resolved": "https://registry.npmjs.org/pprof-format/-/pprof-format-2.2.2.tgz", - "integrity": "sha512-hd90rHVDhNOhgHTmazVzDSVwTLOBjpZQ26AO/0j46sAFZ9uSWY0DcK2zJcwnuo2R6EzufBbOoWlPazA5nSyHcg==", + "version": "2.3.1", + "resolved": "https://registry.npmjs.org/pprof-format/-/pprof-format-2.3.1.tgz", + "integrity": "sha512-y51Z83qG2vEQBACPu6lkGFREVkHwQaCaNDdSFEMLIqSo3bmpADsbP6J3F2SSk7tYB741oTQ9Kt5YAdQsmiCRkA==", "license": "MIT", "optional": true }, "node_modules/source-map": { - "version": "0.7.6", - "resolved": "https://registry.npmjs.org/source-map/-/source-map-0.7.6.tgz", - "integrity": "sha512-i5uvt8C3ikiWeNZSVZNWcfZPItFQOsYTUAOkcUPGd8DqDy1uOUikjt5dG+uRlwyvR108Fb9DOd4GvXfT0N2/uQ==", + "version": "0.8.0", + "resolved": "https://registry.npmjs.org/source-map/-/source-map-0.8.0.tgz", + "integrity": "sha512-d8EqvL+k/SOXCreS/SUzg2ciyHqBBLcN/yuRjFsbvVhHTE2pgei7oAhmPM7kWFbkX6OSMQfUq4KbkF3au9lhYQ==", "license": "BSD-3-Clause", "optional": true, "engines": { diff --git a/test-apps/stock-watchlist-agent-js/package.json b/test-apps/stock-watchlist-agent-js/package.json index 5c5c838e..de80c430 100644 --- a/test-apps/stock-watchlist-agent-js/package.json +++ b/test-apps/stock-watchlist-agent-js/package.json @@ -15,7 +15,7 @@ "node": ">=20" }, "dependencies": { - "dd-trace": "latest", + "dd-trace": "^6.10.0", "dotenv": "^16.4.7", "openai": "^4.104.0" } From d55b6ab2d168f2394e1a65428356c071f0349a92 Mon Sep 17 00:00:00 2001 From: Christopher Fox Date: Fri, 7 Aug 2026 14:18:47 -0400 Subject: [PATCH 3/6] docs(stock-watchlist-agent-js): note where to find image inputs in the trace view Co-Authored-By: Claude Opus 5 --- test-apps/stock-watchlist-agent-js/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/test-apps/stock-watchlist-agent-js/README.md b/test-apps/stock-watchlist-agent-js/README.md index a4fe8202..fcfd3da3 100644 --- a/test-apps/stock-watchlist-agent-js/README.md +++ b/test-apps/stock-watchlist-agent-js/README.md @@ -65,7 +65,9 @@ Arguments ending in `.png`, `.jpg`, `.jpeg`, `.gif`, or `.webp`, plus any `http( `identify_ticker` is annotated as an `llm`-kind span whose user message carries `imageParts: [{ mimeType, content }]`, which the SDK emits as `image_parts: [{ mime_type, content }]`. -This only works with an SDK version that supports image parts, and it must be done by hand: provider auto-instrumentation does not capture images, and `imageParts` is ignored on non-`llm` span kinds (`task`, `workflow`, `agent`), where input is tagged as a plain value. +To see an image input in Datadog, open the trace in **LLM Observability > Traces** and select the span named **`identify_ticker`** — there is one per image, nested under `resolve_tickers_from_images`. The image renders inline in that span's input messages, next to the user question, with the identified ticker in the output. It does not appear on the root `analyze_portfolio` span or on the auto-instrumented `OpenAI.createResponse` child span. + +Requires `dd-trace` 6.10.0 or later. The annotation must also be done by hand: provider auto-instrumentation does not capture images, and `imageParts` is ignored on non-`llm` span kinds (`task`, `workflow`, `agent`), where input is tagged as a plain value. ## Running with Datadog LLM Observability From 541654de70683daf9d5221d69e30edccc18f8f45 Mon Sep 17 00:00:00 2001 From: Christopher Fox Date: Fri, 7 Aug 2026 14:20:03 -0400 Subject: [PATCH 4/6] docs(stock-watchlist-agent-js): trim the image span note Co-Authored-By: Claude Opus 5 --- test-apps/stock-watchlist-agent-js/README.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/test-apps/stock-watchlist-agent-js/README.md b/test-apps/stock-watchlist-agent-js/README.md index fcfd3da3..bb5f17ee 100644 --- a/test-apps/stock-watchlist-agent-js/README.md +++ b/test-apps/stock-watchlist-agent-js/README.md @@ -65,9 +65,7 @@ Arguments ending in `.png`, `.jpg`, `.jpeg`, `.gif`, or `.webp`, plus any `http( `identify_ticker` is annotated as an `llm`-kind span whose user message carries `imageParts: [{ mimeType, content }]`, which the SDK emits as `image_parts: [{ mime_type, content }]`. -To see an image input in Datadog, open the trace in **LLM Observability > Traces** and select the span named **`identify_ticker`** — there is one per image, nested under `resolve_tickers_from_images`. The image renders inline in that span's input messages, next to the user question, with the identified ticker in the output. It does not appear on the root `analyze_portfolio` span or on the auto-instrumented `OpenAI.createResponse` child span. - -Requires `dd-trace` 6.10.0 or later. The annotation must also be done by hand: provider auto-instrumentation does not capture images, and `imageParts` is ignored on non-`llm` span kinds (`task`, `workflow`, `agent`), where input is tagged as a plain value. +To see an image input in Datadog, open the trace in **LLM Observability > Traces** and select the span named **`identify_ticker`**. ## Running with Datadog LLM Observability From 50259b895424212e0a8d5c6a7626e7559b0f671b Mon Sep 17 00:00:00 2001 From: Christopher Fox Date: Fri, 7 Aug 2026 14:24:38 -0400 Subject: [PATCH 5/6] refactor(stock-watchlist-agent-js): support only local image files Drops URL and data-URL image inputs to keep the sample small. Co-Authored-By: Claude Opus 5 --- test-apps/stock-watchlist-agent-js/README.md | 6 ++-- .../src/agents/vision.js | 31 +++---------------- .../stock-watchlist-agent-js/src/main.js | 10 +++--- 3 files changed, 12 insertions(+), 35 deletions(-) diff --git a/test-apps/stock-watchlist-agent-js/README.md b/test-apps/stock-watchlist-agent-js/README.md index bb5f17ee..d3d21855 100644 --- a/test-apps/stock-watchlist-agent-js/README.md +++ b/test-apps/stock-watchlist-agent-js/README.md @@ -53,13 +53,13 @@ npm start -- AAPL GOOGL NVDA # b) First two inputs are images, third is a ticker symbol npm start -- logos/apple.png logos/google.png NVDA -# Image URLs work too, and --image forces an argument to be read as an image -npm start -- https://example.com/nvidia-logo.jpg --image logos/apple.png +# --image forces an argument to be read as an image +npm start -- AAPL --image logos/nvidia.png ``` The `logos/` directory holds small sample wordmark images for Apple, Google, and NVIDIA so the image path can be run without supplying your own files. -Arguments ending in `.png`, `.jpg`, `.jpeg`, `.gif`, or `.webp`, plus any `http(s)://` or `data:image/...` argument, are treated as images automatically. Use `--image ` to force an argument to be read as an image. Every image is read into memory as base64 (URLs are downloaded first) so the same bytes can be sent to OpenAI and attached to the trace. Set `OPENAI_VISION_MODEL` to override the model used for image-to-ticker translation (defaults to `OPENAI_MODEL`, then `gpt-5.4-nano`). +Arguments ending in `.png`, `.jpg`, `.jpeg`, `.gif`, or `.webp` are treated as local image files automatically. Use `--image ` to force an argument to be read as an image. Each image is read as base64 so the same bytes are sent to OpenAI and attached to the trace. Set `OPENAI_VISION_MODEL` to override the model used for image-to-ticker translation (defaults to `OPENAI_MODEL`, then `gpt-5.4-nano`). ### Images on spans diff --git a/test-apps/stock-watchlist-agent-js/src/agents/vision.js b/test-apps/stock-watchlist-agent-js/src/agents/vision.js index 7d66b50b..7f925001 100644 --- a/test-apps/stock-watchlist-agent-js/src/agents/vision.js +++ b/test-apps/stock-watchlist-agent-js/src/agents/vision.js @@ -40,36 +40,13 @@ You identify the publicly traded company shown in an image and return its stock Return JSON only matching the TickerFromImage schema. ` -function isHttpUrl (value) { - return /^https?:\/\//i.test(value) -} - function looksLikeImage (value) { - if (isHttpUrl(value) || value.startsWith('data:image/')) return true return Object.keys(IMAGE_MIME_TYPES).includes(path.extname(value).toLowerCase()) } const IMAGE_QUESTION = 'Which publicly traded company does this image show? Return its ticker symbol.' -async function loadImage (imageInput) { - if (imageInput.startsWith('data:image/')) { - const match = imageInput.match(/^data:(image\/[\w+.-]+);base64,(.*)$/) - if (!match) { - throw new Error(`Malformed image data URL: ${imageInput.slice(0, 32)}...`) - } - return { mimeType: match[1], base64: match[2], source: 'data-url' } - } - - if (isHttpUrl(imageInput)) { - const response = await fetch(imageInput) - if (!response.ok) { - throw new Error(`Failed to download image ${imageInput}: HTTP ${response.status}`) - } - const mimeType = (response.headers.get('content-type') || 'image/png').split(';')[0] - const buffer = Buffer.from(await response.arrayBuffer()) - return { mimeType, base64: buffer.toString('base64'), source: 'url' } - } - +function loadImage (imageInput) { const filePath = path.resolve(imageInput) const extension = path.extname(filePath).toLowerCase() const mimeType = IMAGE_MIME_TYPES[extension] @@ -79,14 +56,14 @@ async function loadImage (imageInput) { if (!fs.existsSync(filePath)) { throw new Error(`Image not found: ${imageInput}`) } - return { mimeType, base64: fs.readFileSync(filePath).toString('base64'), source: 'file' } + return { mimeType, base64: fs.readFileSync(filePath).toString('base64') } } async function identifyTicker (imageInput, model = DEFAULT_VISION_MODEL) { // Annotated as an `llm` span with imageParts: the LLMObs SDK only renders images // on manually annotated llm-kind messages, not from provider auto-instrumentation. return traceSpan({ kind: 'llm', name: 'identify_ticker', modelName: model, modelProvider: 'openai' }, async span => { - const { mimeType, base64, source } = await loadImage(imageInput) + const { mimeType, base64 } = loadImage(imageInput) annotate(span, { inputData: [ { role: 'system', content: VISION_PROMPT }, @@ -96,7 +73,7 @@ async function identifyTicker (imageInput, model = DEFAULT_VISION_MODEL) { imageParts: [{ mimeType, content: base64 }], }, ], - metadata: { model, image_source: source, image_input: imageInput }, + metadata: { model, image_input: imageInput }, }) const response = await client.responses.create({ diff --git a/test-apps/stock-watchlist-agent-js/src/main.js b/test-apps/stock-watchlist-agent-js/src/main.js index 0e321f0b..eb870021 100755 --- a/test-apps/stock-watchlist-agent-js/src/main.js +++ b/test-apps/stock-watchlist-agent-js/src/main.js @@ -9,14 +9,14 @@ const { runEvaluations } = require('./evals') const USAGE = [ 'Usage: node src/main.js [TICKER|IMAGE ...]', '', - 'Inputs may be ticker symbols, image files, or image URLs (mixed freely).', + 'Inputs may be ticker symbols or local image files (mixed freely).', 'Images are translated to ticker symbols by an LLM before research begins.', - 'Use --image to force an argument to be treated as an image.', + 'Use --image to force an argument to be treated as an image.', '', 'Examples:', ' node src/main.js AAPL GOOGL NVDA', - ' node src/main.js ./logos/apple.png https://example.com/nvidia-logo.jpg', - ' node src/main.js AAPL --image ./logos/tesla.png', + ' node src/main.js logos/apple.png logos/google.png NVDA', + ' node src/main.js AAPL --image logos/nvidia.png', ].join('\n') function printBriefing (briefing) { @@ -81,7 +81,7 @@ function parseArgs (argv) { if (arg === '--image') { const value = args[++i] if (!value) { - throw new Error('--image requires a file path or URL') + throw new Error('--image requires a file path') } images.push(value) } else if (looksLikeImage(arg)) { From 7ab3bc799ada048a0920bf3bbc6b78616d2a1335 Mon Sep 17 00:00:00 2001 From: Christopher Fox Date: Fri, 7 Aug 2026 14:35:12 -0400 Subject: [PATCH 6/6] refactor(stock-watchlist-agent-js): drop OPENAI_VISION_MODEL in favor of OPENAI_MODEL Co-Authored-By: Claude Opus 5 --- test-apps/stock-watchlist-agent-js/.env.example | 2 -- test-apps/stock-watchlist-agent-js/README.md | 2 +- test-apps/stock-watchlist-agent-js/src/agents/vision.js | 4 ++-- 3 files changed, 3 insertions(+), 5 deletions(-) diff --git a/test-apps/stock-watchlist-agent-js/.env.example b/test-apps/stock-watchlist-agent-js/.env.example index 42b6479e..33c37256 100644 --- a/test-apps/stock-watchlist-agent-js/.env.example +++ b/test-apps/stock-watchlist-agent-js/.env.example @@ -4,8 +4,6 @@ OPENAI_API_KEY=sk-... # OPENAI_MODEL=gpt-5.4-nano # OPENAI_SEARCH_MODEL=gpt-5.4-nano # OPENAI_EVAL_MODEL=gpt-5.4-nano -# Model used to translate image inputs into ticker symbols -# OPENAI_VISION_MODEL=gpt-5.4-nano # Datadog LLM Observability DD_API_KEY= diff --git a/test-apps/stock-watchlist-agent-js/README.md b/test-apps/stock-watchlist-agent-js/README.md index d3d21855..b05c6896 100644 --- a/test-apps/stock-watchlist-agent-js/README.md +++ b/test-apps/stock-watchlist-agent-js/README.md @@ -59,7 +59,7 @@ npm start -- AAPL --image logos/nvidia.png The `logos/` directory holds small sample wordmark images for Apple, Google, and NVIDIA so the image path can be run without supplying your own files. -Arguments ending in `.png`, `.jpg`, `.jpeg`, `.gif`, or `.webp` are treated as local image files automatically. Use `--image ` to force an argument to be read as an image. Each image is read as base64 so the same bytes are sent to OpenAI and attached to the trace. Set `OPENAI_VISION_MODEL` to override the model used for image-to-ticker translation (defaults to `OPENAI_MODEL`, then `gpt-5.4-nano`). +Arguments ending in `.png`, `.jpg`, `.jpeg`, `.gif`, or `.webp` are treated as local image files automatically. Use `--image ` to force an argument to be read as an image. Each image is read as base64 so the same bytes are sent to OpenAI and attached to the trace. Image-to-ticker translation uses `OPENAI_MODEL` (defaults to `gpt-5.4-nano`), so that model must support image inputs. ### Images on spans diff --git a/test-apps/stock-watchlist-agent-js/src/agents/vision.js b/test-apps/stock-watchlist-agent-js/src/agents/vision.js index 7f925001..7e8002a3 100644 --- a/test-apps/stock-watchlist-agent-js/src/agents/vision.js +++ b/test-apps/stock-watchlist-agent-js/src/agents/vision.js @@ -8,7 +8,7 @@ const { jsonSchemaFormat } = require('./responses-agent') const OpenAI = require('openai') const client = new OpenAI() -const DEFAULT_VISION_MODEL = process.env.OPENAI_VISION_MODEL || process.env.OPENAI_MODEL || 'gpt-5.4-nano' +const DEFAULT_MODEL = process.env.OPENAI_MODEL || 'gpt-5.4-nano' const IMAGE_MIME_TYPES = { '.png': 'image/png', @@ -59,7 +59,7 @@ function loadImage (imageInput) { return { mimeType, base64: fs.readFileSync(filePath).toString('base64') } } -async function identifyTicker (imageInput, model = DEFAULT_VISION_MODEL) { +async function identifyTicker (imageInput, model = DEFAULT_MODEL) { // Annotated as an `llm` span with imageParts: the LLMObs SDK only renders images // on manually annotated llm-kind messages, not from provider auto-instrumentation. return traceSpan({ kind: 'llm', name: 'identify_ticker', modelName: model, modelProvider: 'openai' }, async span => {