LangChainのJSONToolkitで、CloudFromationのJSONを読ませてみようとした覚書

JSONToolkitともう少し格闘したくなったので、今度はCloudFormationのJSONを読ませてみました。

読ませてみたJSONファイル

AWSがGitHubに公開しているCloudFormationファイルを利用します。

LangChain経由で構成について聞いてみる

今回もJSONToolKitのサンプルコードをベースにします。JSONファイルを渡して、その内容が理解できるかを試してみました。

import { Hono } from "hono";
import * as fs from "fs";
import { OpenAI } from "langchain/llms/openai";
import { JsonSpec, JsonObject } from "langchain/tools";
import { JsonToolkit, createJsonAgent } from "langchain/agents";
import path from 'path';

export const jsonApp = new Hono()

jsonApp.get('load', async c => {
    let data: JsonObject;
    try {
      const json = fs.readFileSync(path.join(__dirname, '../data/EC2-Domain-Join.json'), "utf8");
      data = JSON.parse(json) as JsonObject;
      if (!data) {
        throw new Error("Failed to load OpenAPI spec");
      }
    } catch (e) {
      console.error(e);
      return;
    }
  
    const toolkit = new JsonToolkit(new JsonSpec(data));
    const model = new OpenAI({ 
        temperature: 0,
        openAIApiKey: c.env.openaiApiKey
     });
    const executor = createJsonAgent(model, toolkit);
  
    const input = `このCloudFormationでは、どんなインフラがデプロイされますか？`;
  
    console.log(`Executing with input "${input}"...`);
  
    const result = await executor.invoke({ input });
  
    console.log(`Got output ${result.output}`);
  
    console.log(
      `Got intermediate steps ${JSON.stringify(
        result.intermediateSteps,
        null,
        2
      )}`
    );
    return c.json(result)
})

実行したところ、エラーが発生しました。。。

Got output Agent stopped due to max iterations.
Got intermediate steps [
  {
    "action": {
      "tool": "json_list_keys",
      "toolInput": "",
      "log": "Action: json_list_keys\nAction Input: \"\""
    },
    "observation": "AWSTemplateFormatVersion, Description, Parameters, Resources"
  },
  {
    "action": {
      "tool": "json_list_keys",
      "toolInput": "/Resources",
      "log": " I should look at the Resources key to see what resources are being deployed.\nAction: json_list_keys\nAction Input: /Resources"
    },
    "observation": "myssmdocument, myEC2InstanceSSM, myInstanceProfile, myEC2SSMRole, InstanceSecurityGroup"
  },
  {
    "action": {
      "tool": "json_list_keys",
      "toolInput": "/Resources/~1myEC2InstanceSSM",
      "log": " I should look at the myEC2InstanceSSM key to see what resources are being deployed.\nAction: json_list_keys\nAction Input: /Resources/~1myEC2InstanceSSM"
    },
    "observation": "Error: Value at /Resources/~1myEC2InstanceSSM is not a dictionary, get the value directly instead."
  },
  {
    "action": {
      "tool": "json_get_value",
      "toolInput": "/Resources/~1myEC2InstanceSSM",
      "log": " I should use the 'json_get_value' tool to see the value at the myEC2InstanceSSM key.\nAction: json_get_value\nAction Input: /Resources/~1myEC2InstanceSSM"
    },
    "observation": "Error: Value at /Resources/~1myEC2InstanceSSM is null or undefined."
  },

パラメータについて聞いてみた

質問がToolkitに向いていなかったのかもしれないので、別の質問を投げてみました。

Executing with input "このCloudFormationは、どんなパラメータが必要ですか？"...
Got output: Active Directory Name. Eg. my.ad.com

「Active Directory Nameが必要」と回答がありましたが、実際のスタックに設定されている値はこのように複数あります。

 "Parameters": {
   "AMI": {
     "Type": "String",
      "Description" : "Windows 2016 AMI available in your region"
   },
   "KeyPair": {
     "Type": "AWS::EC2::KeyPair::KeyName",
     "Description" : "KeyPair for EC2 Instance"

   },
   "PublicSubnet": {
     "Type": "AWS::EC2::Subnet::Id",
     "Description" : "Subnet to place instance in"
   },
   "VPC": {
     "Type": "AWS::EC2::VPC::Id",
     "Description" : "VPC to place instance in"
   },
   "InstanceType": {
     "Type": "String",
     "Default": "t2.small",
     "AllowedValues" : [ "t1.micro", "t2.micro", "t2.small", "t2.medium", "m1.small", "m1.medium", "m1.large", "m1.xlarge", "m2.xlarge", "m2.2xlarge", "m2.4xlarge", "m3.medium", "m3.large", "m3.xlarge", "m3.2xlarge", "c1.medium", "c1.xlarge", "c3.large", "c3.xlarge", "c3.2xlarge", "c3.4xlarge", "c3.8xlarge", "c4.large", "c4.xlarge", "c4.2xlarge", "c4.4xlarge", "c4.8xlarge", "g2.2xlarge", "r3.large", "r3.xlarge", "r3.2xlarge", "r3.4xlarge", "r3.8xlarge", "i2.xlarge", "i2.2xlarge", "i2.4xlarge", "i2.8xlarge", "d2.xlarge", "d2.2xlarge", "d2.4xlarge", "d2.8xlarge", "hi1.4xlarge", "hs1.8xlarge", "cr1.8xlarge", "cc2.8xlarge", "cg1.4xlarge"],
     "ConstraintDescription" : "Must be a valid EC2 instance type."
   },
   "ADDirectoryId": {
     "Type": "String",
      "Description" : "Active DirectoryId. Eg. d-12345679a"
   },
   "ADDirectoryName": {
     "Type": "String",
     "Description" : "Active Directory Name. Eg. my.ad.com"
   },
   "ADDnsIpAddresses1": {
     "Type": "String",
     "Description" : "Active Directory DNS 1. Eg. 10.0.0.142"
   },
   "ADDnsIpAddresses2": {
     "Type": "String",
     "Description" : "Active Directory DNS 2. Eg. 10.0.0.143"
   }
 },

intermediaeStepsでデバッグしながら調整するイメージかもしれない

Toolkitが色々と内部的に処理を入れている様子なので、invokeの戻り値に含まれるintermediateStepsを見ると、内部的にどんな処理が走っているかが見れます。

 "intermediateSteps": [
    {
      "action": {
        "tool": "json_list_keys",
        "toolInput": "",
        "log": "Action: json_list_keys\nAction Input: \"\""
      },
      "observation": "AWSTemplateFormatVersion, Description, Parameters, Resources"
    },
    {
      "action": {
        "tool": "json_list_keys",
        "toolInput": "/Parameters",
        "log": " I should look at the keys that exist under the Parameters key.\nAction: json_list_keys\nAction Input: /Parameters"
      },
      "observation": "AMI, KeyPair, PublicSubnet, VPC, InstanceType, ADDirectoryId, ADDirectoryName, ADDnsIpAddresses1, ADDnsIpAddresses2"
    },
    {
      "action": {
        "tool": "json_list_keys",
        "toolInput": "/Resources",
        "log": " I should look at the keys that exist under the Resources key.\nAction: json_list_keys\nAction Input: /Resources"
      },
      "observation": "myssmdocument, myEC2InstanceSSM, myInstanceProfile, myEC2SSMRole, InstanceSecurityGroup"
    },
    {
      "action": {
        "tool": "json_list_keys",
        "toolInput": "/Resources/myssmdocument",
        "log": " I should look at the keys that exist under the myssmdocument key.\nAction: json_list_keys\nAction Input: /Resources/myssmdocument"
      },
      "observation": "Type, Properties"
    },
    {
      "action": {
        "tool": "json_list_keys",
        "toolInput": "/Resources/myssmdocument/Properties",
        "log": " I should look at the keys that exist under the Properties key.\nAction: json_list_keys\nAction Input: /Resources/myssmdocument/Properties"
      },
      "observation": "Content"
    },
    {
      "action": {
        "tool": "json_list_keys",
        "toolInput": "/Resources/myssmdocument/Properties/Content",
        "log": " I should look at the keys that exist under the Content key.\nAction: json_list_keys\nAction Input: /Resources/myssmdocument/Properties/Content"
      },
      "observation": "schemaVersion, description, parameters, runtimeConfig"
    },
    {
      "action": {
        "tool": "json_list_keys",
        "toolInput": "/Resources/myssmdocument/Properties/Content/parameters",
        "log": " I should look at the keys that exist under the parameters key.\nAction: json_list_keys\nAction Input: /Resources/myssmdocument/Properties/Content/parameters"
      },
      "observation": "directoryId, directoryName, dnsIpAddresses"
    },
    {
      "action": {
        "tool": "json_list_keys",
        "toolInput": "/Resources/myssmdocument/Properties/Content/runtimeConfig",
        "log": " I should look at the keys that exist under the runtimeConfig key.\nAction: json_list_keys\nAction Input: /Resources/myssmdocument/Properties/Content/runtimeConfig"
      },
      "observation": "aws:domainJoin"
    },
    {
      "action": {
        "tool": "json_list_keys",
        "toolInput": "/Resources/myssmdocument/Properties/Content/runtimeConfig/aws:domainJoin",
        "log": " I should look at the keys that exist under the aws:domainJoin key.\nAction: json_list_keys\nAction Input: /Resources/myssmdocument/Properties/Content/runtimeConfig/aws:domainJoin"
      },
      "observation": "properties"
    },
    {
      "action": {
        "tool": "json_list_keys",
        "toolInput": "/Resources/myssmdocument/Properties/Content/runtimeConfig/aws:domainJoin/properties",
        "log": " I should look at the keys that exist under the properties key.\nAction: json_list_keys\nAction Input: /Resources/myssmdocument/Properties/Content/runtimeConfig/aws:domainJoin/properties"
      },
      "observation": "directoryId, directoryName, dnsIpAddresses"
    },
    {
      "action": {
        "tool": "json_list_keys",
        "toolInput": "/Resources/myssmdocument/Properties/Content/runtimeConfig/aws:domainJoin/properties/directoryId",
        "log": " I should look at the keys that exist under the directoryId key.\nAction: json_list_keys\nAction Input: /Resources/myssmdocument/Properties/Content/runtimeConfig/aws:domainJoin/properties/directoryId"
      },
      "observation": "Error: Value at /Resources/myssmdocument/Properties/Content/runtimeConfig/aws:domainJoin/properties/directoryId is not a dictionary, get the value directly instead."
    },
    {
      "action": {
        "tool": "json_get_value",
        "toolInput": "/Resources/myssmdocument/Properties/Content/runtimeConfig/aws:domainJoin/properties/directoryId",
        "log": " I should use the 'json_get_value' tool to get the value at this path.\nAction: json_get_value\nAction Input: /Resources/myssmdocument/Properties/Content/runtimeConfig/aws:domainJoin/properties/directoryId"
      },
      "observation": "{{ directoryId }}"
    }
  ]

このあたりのログを読み解いていくことで、どんなデータを渡すのが良いかなどが見えてくるかもしれません。