The 2026 Shanghai Open Source Software Application Innovation Competition is hosted byOpen Source Chinaand is an important event implementing the "Shanghai Implementation Plan for Strengthening Open Source System Construction," aimed at promoting the deep integration of AI and the software industry and supporting Shanghai's strategic goal of becoming a "Global Open Source City."

As the2026 Shanghai Open Source Software Application Innovation Competitionofficially kicks off, the OpenAtom openKylin (hereinafter referred to as "openKylin") community has released a majorexclusive competition topic for the "Open Source AI Tools Track" —"Design of an Automated Evaluation Benchmark for Long-Term Memory of Intelligent Agents for the openKylin Operating System",calling for high-quality solutions from open source developers nationwide to create a standardized, practical evaluation "yardstick" for long-term agent memory capabilities.

openKylin currently supports multiple agent frameworks including KylinBot, kylin-agent, OpenClaw,Hermes Agent and others, and requires a matching system-level automated evaluation tool to quantitatively compare the memory capabilities of different agents on the openKylin operating system. This is the core starting point of this competition topic.

This exclusive openKylin topic precisely focuses on three major pain points in the current evaluation of large-model agent memory and builds a complete evaluation solution accordingly:

First, current evaluations often rely on manual inspection, making them difficult to scale; second, simple Q&A-style verification items cannot cover conflict updates, similar interference, and action reuse in real usage; third, automated scoring is prone to semantic drift, making it difficult to stably determine whether a model has "remembered correctly," "misremembered," or "remembered something it should not have."

Based on this, the topic requires participants to design anautomated, low-manual-intervention, reproducible, and extensibleevaluation solution in the openKylin environment, organizing conversations, memory records, action trajectories, files, and other data as unified evidence to automatically assess agents' long-term memory capabilities, focusing on six capabilities: long-term retention, memory recall, dynamic updates, similar distinction, boundary recognition, and task reuse.

In terms of deliverables, participating teams must provide complete solution documentation covering test objectives, data generation methods, validation case design, and automated scoring processes. The dataset design must provide extensible data structures, and automated evaluation must output explainable scoring reasons with minimal manual intervention. Teams must also provide sample data and results, reproducible materials (code must compile and run on openKylin), and package the evaluation process as a one-click CLI tool, with a .deb software package recommended. The demonstration must run evaluations on at least two agents in the openKylin desktop environment and produce a 3-5 minute full screen recording with a multi-dimensional comparison radar chart.

This topic will follow the competition's official evaluation criteria, scoring from six core dimensions to comprehensively verify the professionalism, innovation, and practicality of solutions:

• (15%) Task definition and generality — whether it can evaluate different agents and evidence sources and integrate well with the openKylin architecture

• (25%) Data design quality — sample coverage and discriminative power

• (25%) Automated scoring capability — whether it can reduce manual review and distinguish correct memory from erroneous reuse

• (15%) Stability and reproducibility

• (10%) Metric completeness — requiring multi-dimensional metric output

• (10%) Innovation and engineering practicality

This topic is suitable for teams and individuals with experience inLLMapplication development, agent construction, and AI evaluation system development. University teams, research institutions, open source community developers, and enterprise technical teams are all welcome to register.

The registration and submission deadline isOctober 11, 2026, and the offline finals will be held inShanghai on October 24. Participating teams will have the opportunity to compete for the competition's total prize pool of over one million yuan, computing tokens, large-model APIs, and other exclusive developer resources. Outstanding projects will also be included in the competition's official project library, receiving supporting services such as industry matchmaking, technical validation, investment and financing matchmaking, and policy consultation, with the opportunity to be selected for Shanghai's open source project cultivation program. Open source technology enthusiasts nationwide are welcome to register!

openKylin invites open source technology enthusiasts nationwide to join the competition, jointly improve the open source operating system agent evaluation system, and help open source AI technology take root and thrive!

You can also click"Read Original" to visit the competition's official website for more details:

https://www.oschina.net/os2026/