Created by @chuxij & @chatgpt

Please See Release:

https://github.com/timedomain-tech/open-creator

Updates

timestamp comments
2023-09-14 the first draft was created

update skill extractor agent’s prompt and schema also test on open-interpreter, see: https://github.com/KillianLucas/open-interpreter/pull/399

notice: the implementation may be different with this blog the ideas of this will be iteratively updated | | 2023-09-18 | to have a bigger picture,

supplement for skill object/interface:

about cli:

about test envrioment

  1. Running in Docker
  2. Creating a separate conda environment for each skill (since conda caches can be reused) This ensures security and resolves dependency conflicts. However, Docker isolates local directories, and users need to install conda. This can make it less lightweight. If a virtual environment is solely needed, venv can also be an option. We can let users choose between conda and venv later on, but Docker might not be feasible. The earlier Docker solution discussion was about having a virtual partition, like with GPT-4, where files reside. This topic can be open for discussion and added to the roadmap for developer contributions. A dedicated channel on Discord would be perfect. Regarding Langchain's dependency, the reason I wanted to eliminate it is that some developers dislike using Langchain. We previously felt that Langchain was too heavily wrapped. It's great for direct project use, but as a library, it might deter some users. Why not implement the core functionalities of Langchain ourselves? But then the workload would increase considerably. This might make our code less clear, and we'd end up wrapping it like Langchain. Such legacy code is hard for people to study, leading to pull requests and issues. I believe the first version should be as lightweight as possible. HuggingFace Standards:
  1. Use list_files_info and then choose skill
  2. Use hf_hub_download specifying the skill name
  3. Ensure there's a skill.json file within user/open-creator-skill-library/<skill_name> Other Thoughts and Reminders:
  4. Support for HuggingFace, Git, and Langchain hubs
  5. Provide three usage examples in the documentation
  6. Multiple ways to create skills, with support for community hub imports and uploads
  7. Persistent skills in an interpreter context, with support for saving, importing, and searching
  8. More generalized use-cases, converting classic Langchain agents, HuggingFace models, and frequent tasks (like file handling, data analysis, plotting, web scraping) into skills. Viewing open-creator itself as a skill.
  9. Best practices for executing code skills: automatic dependency checks and installations during skill installation, input type verification using Pydantic, and AST package for execution.
  10. Clear prompts about standards: doc strings, production-grade code, readability, low complexity, and adhering to the best practices of the current language. The code should resemble the formality of the Transformers library.
  11. Add an additional args dictionary in metadata for new fields and annotations.
  12. For remote libraries, auto-generate READMEs and usage instructions. Allow users to copy open-creator code with a single click.

Technology Report Name: open creator: filling the gap between code interpreter and skill library | | 2023-09-21 | Current Progress

We are setting up a repository on Space named skill-library. Inside it, we will store the following:

-- Normally, we want to create a skill object after an open-interpreter session (see the pull request 399) . Thanks to the skill_extractor_agent, we can use our conversation history and convert it into a formatted skill object by using %save_skill command. But now, we can create a skill object by using only a request.

How this happened? We fork the code of open-interpreter , refactor and re-implement it into a minimal version Code interpreter works in only 45 lines has been integrated into open-creatorpython import subprocess import traceback def get_persistent_process(start_cmd: str): process = subprocess.Popen( args=start_cmd.split(), stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, bufsize=0 ) return process class BaseInterpreter: """A tool for running base code in a terminal.""" name: str = "base_interpreter" description: str = ( "A base shell command tool. Use this to execute bash commands. " "It can also be used to execute any language with interactive mode" ) def __init__(self): self.process = None def run(self, query: str, is_start: bool = False) -> dict: if is_start or self.process is None: try: self.process = get_persistent_process(query) return {"status": "success", "stdout": "", "stderr": ""} except Exception: traceback_string = traceback.format_exc() return {"status": "error", "stdout": "", "stderr": traceback_string} stdout, stderr = "", "" try: stdout, stderr = self.process.communicate(input=query) except BrokenPipeError: stderr = traceback.format_exc() return {"status": "success", "stdout": stdout, "stderr": stderr} bash (open_creator_dev) ➜ code_interpreter git:(main) ✗ tree . . ├── R.py ├── __init__.py ├── applescript.py ├── base.py ├── html.py ├── javascript.py ├── julia.py ├── python.py └── shell.py

Like chatDev, GPT-Enginer, GPT-team and etc, we can use autonomous agent. There will be agents not only help you write code, converting codes into skills, but also doing the tests, and the refactor stuff... -— | | | |

Untitled

Why the Need?

Scalability

GPT-4 naturally leans towards providing detailed steps and explanations. However, in situations with limited context, addressing intricate scenarios requires abstract skill libraries. Users must laboriously assemble these domain-specific operations. A pitfall here is that these meticulously organized skill sets aren't easily reusable. This makes tasks like constructing intricate, layered projects a significant challenge.

Cost-Effective

Often, achieving the right script or code necessitates multiple iterations. These very scripts or codes are then frequently invoked in our daily routines, say, checking upcoming birthdays. What we need is a skill_librarymechanism that allows us to consolidate and archive the refined versions of these codes, turning them into readily usable skill sets. So, the next time around, we can just call upon this package instead of iterating all over again to derive the correct script. This not only saves us time but also conserves tokens.

Community Wisdom Untapped

A significant shortcoming of lacking a cohesive "skill_library" is the missed opportunity to harness the collective intelligence of the global community. All over the world, ingenious developers and users continuously discover optimized solutions to challenges. Without a centralized platform to archive and share these insights, the wheel gets reinvented repeatedly. A skill library could serve as a repository where community members contribute, refine, and validate diverse solutions, amplifying shared knowledge's potential.

Consistency and Enhanced Robustness

A dedicated "skill_library" ensures users experience consistency when tackling challenges. Accessing well-curated and polished knowledge not only offers reliable solutions but also promises uniform outcomes. This uniformity becomes vital when reflecting on the frustrations tied to replicating another person's successful process. Erratic or unpredictable experiences can be exasperating. A standardized skill library offers robust solutions, eliminating the inconsistencies often associated with problem-solving.

How to Implement It?

In the vast seascape of problem-solving mechanisms, we have an array of potential solutions. Some are already in play, while others remain as budding concepts, ripe for exploration.