Llm-provider Module Design and Implementation

This note records the design and API implementation of the llm-provider module in the interview-guide project. This module is responsible for unified management of large model Provider configuration, including model lists, default models, Embedding capability, connectivity testing, and ASR/TTS runtime configuration for voice interviews.

Module Capability Overview

• Provider management: supports querying, creating, updating, and deleting LLM Providers.
• Dual storage modes: supports both DB mode and Legacy configuration-file mode.
• Secret protection: in DB mode, API Keys are encrypted with AES-GCM and masked before being returned by APIs.
• Default model management: separates the default Chat Provider from the default Embedding Provider.
• Cache reload: after Provider changes, clears ChatClient and EmbeddingModel caches and rebuilds them on next use.
• Embedding validation: validates model type, dimensions, and capability switches when creating, updating, or setting the default Embedding Provider.
• Connectivity testing: supports sending real HTTP test requests to LLM Providers.
• Voice configuration management: supports reading and updating Qwen ASR/TTS configuration and reloading runtime services.

Flowchart

stateDiagram-v2
  [*] --> ProviderConfigPage: Open LLM Provider config page

  ProviderConfigPage --> ProviderListLoaded: GET /api/llm-provider/list
  ProviderListLoaded --> ProviderDetailLoaded: GET /api/llm-provider/{id}

  ProviderListLoaded --> CreatingProvider: POST /api/llm-provider
  CreatingProvider --> ProviderSaved: Validation passed and saved
  CreatingProvider --> Error: duplicated id / invalid params / write failure

  ProviderDetailLoaded --> UpdatingProvider: PUT /api/llm-provider/{id}
  UpdatingProvider --> ProviderSaved: Partial fields updated
  UpdatingProvider --> Error: Provider missing / invalid params / write failure

  ProviderSaved --> RegistryReloaded: registry.reload()
  RegistryReloaded --> ProviderListLoaded: Reload list

  ProviderDetailLoaded --> TestingProvider: POST /api/llm-provider/{id}/test
  TestingProvider --> TestSuccess: External LLM API connected
  TestingProvider --> TestFailed: connection failed / auth failed / model unavailable
  TestSuccess --> ProviderDetailLoaded
  TestFailed --> ProviderDetailLoaded

  ProviderDetailLoaded --> UpdatingDefaultChat: PUT /api/llm-provider/default-provider
  UpdatingDefaultChat --> DefaultChatUpdated: Provider exists
  UpdatingDefaultChat --> Error: Provider missing / defaultProvider empty

  ProviderDetailLoaded --> UpdatingDefaultEmbedding: PUT /api/llm-provider/default-embedding-provider
  UpdatingDefaultEmbedding --> DefaultEmbeddingUpdated: Embedding supported
  UpdatingDefaultEmbedding --> Error: Provider missing / Embedding unsupported

  DefaultChatUpdated --> RegistryReloaded
  DefaultEmbeddingUpdated --> RegistryReloaded

  ProviderDetailLoaded --> DeletingProvider: DELETE /api/llm-provider/{id}
  DeletingProvider --> ProviderDeleted: Not a default Provider
  DeletingProvider --> Error: Provider missing / default Provider cannot be deleted
  ProviderDeleted --> RegistryReloaded

  ProviderListLoaded --> ManualReloading: POST /api/llm-provider/reload
  ManualReloading --> RegistryReloaded

  ProviderConfigPage --> VoiceConfigLoaded: GET /voice/asr or GET /voice/tts
  VoiceConfigLoaded --> UpdatingVoiceConfig: PUT /voice/asr or PUT /voice/tts
  UpdatingVoiceConfig --> VoiceConfigSaved: Write YAML and reload ASR/TTS service
  UpdatingVoiceConfig --> Error: config write failure

  VoiceConfigLoaded --> TestingAsr: POST /voice/asr/test
  TestingAsr --> AsrTestSuccess: WebSocket port connected
  TestingAsr --> AsrTestFailed: connection failed
  AsrTestSuccess --> VoiceConfigLoaded
  AsrTestFailed --> VoiceConfigLoaded

  Error --> ProviderConfigPage: Show error and return to config page

Core Design

The core of the llm-provider module is managing model configuration reads, secret protection, default model selection, and runtime client caches in one service.

In DB mode, Provider configuration comes from the database. After the service reads LlmProviderEntity, it decrypts the API Key, masks it, and converts the result into ProviderDTO for the frontend. The plaintext API Key only appears briefly at runtime on the server side and is never returned by the API.

In Legacy mode, Provider configuration comes from ConfigurationProperties. Create, update, and delete operations also modify the YAML configuration file and .env file, then reload the Provider registry after the update.

The module uses rwLock to control concurrent reads and writes. Query APIs use a read lock, while create, update, delete, and default-value updates use a write lock to avoid inconsistent configuration during concurrent access.

Provider List Query

GET /api/llm-provider/list Get All Providers

Returns:

• Result<List<ProviderDTO>>

Call chain:

providerController.listProviders();
providerService.listProviders();
globalSettingRepository.findById(1L);
providerRepository.findAll();
encryptionService.decrypt(nonce, ciphertext);

Flow:

1. Controller calls listProviders().
2. Service acquires rwLock.readLock().
3. In DB mode, it first queries global settings to identify the default Chat Provider and default Embedding Provider.
4. It queries all LlmProviderEntity records.
5. It iterates over each Provider:
   • Decrypts the API Key.
   • Calls maskApiKey(...) to mask it.
   • Calls resolveEmbeddingDimensions(...) to resolve vector dimensions, using the global default when not configured.
   • Maps it to ProviderDTO.
6. In Legacy mode, it reads in-memory configuration from properties.getProviders().
7. It returns the Provider list.

Key points:

• DB read failures throw BusinessException(PROVIDER_CONFIG_READ_FAILED).
• API Keys are never returned to the frontend in plaintext.
• There is a current issue: if DB storage is enabled for LLM configuration, changes to configuration files and API Keys will not automatically sync to DB even after restarting the project, unless DB mode is disabled or the database configuration is cleaned.

GET /api/llm-provider/{id} Get a Single Provider

Returns:

• Result<ProviderDTO>

Flow:

1. Controller receives the Provider id.
2. Service acquires the read lock.
3. In DB mode, it queries global settings and the target Provider.
4. If the Provider does not exist, it throws BusinessException(PROVIDER_NOT_FOUND).
5. It decrypts the API Key and masks it.
6. It resolves Embedding dimensions and builds ProviderDTO.
7. In Legacy mode, it gets the Provider by id from in-memory configuration.

Provider Creation and Update

POST /api/llm-provider Create a Provider

Returns:

• Result<Void>

Call chain:

providerService.createProvider(request);
providerRepository.existsById(request.id());
validateEmbeddingConfig(...);
encryptionService.encrypt(apiKey);
providerRepository.save(entity);
registry.reload();

Flow:

1. Controller receives CreateProviderRequest.
2. @Valid ensures id, baseUrl, apiKey, and model are not blank.
3. Service starts a transaction and acquires the write lock.
4. In DB mode, it first checks whether the Provider ID already exists.
5. It performs secondary non-blank validation on baseUrl, model, and apiKey.
6. It calls validateEmbeddingConfig(...) to validate Embedding configuration.
7. It encrypts the API Key with encryptionService.encrypt(apiKey).
8. It saves LlmProviderEntity.
9. It calls registry.reload() to clear runtime caches.

Legacy mode handling:

• Checks whether properties.getProviders() already contains the same ID.
• Builds ProviderConfig and puts it into the in-memory Map.
• Calls writeProviderToYaml(...) to write back to YAML.
• Calls writeEnvValue(...) to write to .env.
• Calls registry.reload() to reload caches.

Embedding validation logic:

supportsEmbedding = true
embeddingModel == null        // throw error
looksLikeChatModel(...)       // throw error and recommend a model
embeddingDimensions <= 0      // throw error

PUT /api/llm-provider/{id} Update a Provider

Returns:

• Result<Void>

Call chain:

providerService.updateProvider(id, request);
providerRepository.findById(id);
validateEmbeddingConfig(...);
encryptionService.encrypt(newApiKey);
providerRepository.save(entity);
registry.reload();

Flow:

1. Controller receives Provider id and UpdateProviderRequest.
2. Service starts a transaction and acquires the write lock.
3. In DB mode, it queries the Provider by id.
4. If the Provider does not exist, it throws BusinessException(PROVIDER_NOT_FOUND).
5. It updates fields selectively:
   • baseUrl: null means no update; empty string is illegal.
   • model: null means no update; empty string is illegal.
   • apiKey: null means no update; empty string is illegal; re-encrypted when updated.
   • embeddingModel: can be null to clear it.
   • embeddingDimensions: updated from the request value.
   • supportsEmbedding: updated from the request value.
   • temperature: updated from the request value.
6. It calls validateEmbeddingConfig(...) for full validation.
7. It saves the entity and reloads caches.

Notes:

• UpdateProviderRequest does not use @Valid, and all fields are optional.
• null means do not update.
• Empty strings are treated as invalid input.

Provider Deletion and Reload

DELETE /api/llm-provider/{id} Delete a Provider

Returns:

• Result<Void>

Flow:

1. Service starts a transaction and acquires the write lock.
2. In DB mode, it reads global settings.
3. It checks whether the current Provider is the default Chat Provider or default Embedding Provider.
4. If it is a default Provider, it throws BusinessException(PROVIDER_DEFAULT_CANNOT_DELETE).
5. It queries the target Provider, confirms it exists, and deletes it.
6. It calls registry.reload() to clear runtime caches.

Legacy mode handling:

• Checks whether the Provider is the default Provider.
• Removes the configuration from the in-memory Map.
• Calls removeProviderFromYaml(...) to remove the YAML node.
• Calls removeFromEnv(...) to remove the API Key line from .env.
• Calls registry.reload() to reload caches.

Protection mechanism:

• Default Chat Provider and default Embedding Provider cannot be deleted directly.
• The default value must be switched before deleting the original Provider.

POST /api/llm-provider/reload Manually Reload Provider Cache

Returns:

• Result<Void>

Logic:

registry.reload();
clientCache.clear();
embeddingModelCache.clear();

Notes:

• This API does not acquire a lock.
• It does not start a transaction.
• It does not access the database.
• It only clears in-memory ChatClient and EmbeddingModel caches.
• The next call to getChatClient() or Embedding model retrieval rebuilds clients from the latest configuration.

Provider Connectivity Test

POST /api/llm-provider/{id}/test Test Provider Connection

Returns:

• Result<ProviderTestResult>

Flow:

1. Service acquires the read lock.
2. It reads runtime configuration based on the current mode:
   • In DB mode, calls getProviderRuntimeConfigOrThrow(id).
   • In Legacy mode, calls toRuntimeConfig(...).
3. It builds a RestClient:
   • connectTimeout = 5s
   • readTimeout = 10s
   • Header: Authorization: Bearer {apiKey}
4. It builds the test request body:

{
  "model": "xxx",
  "messages": [
    {
      "role": "user",
      "content": "Reply with OK only."
    }
  ],
  "max_tokens": 1
}

5. It builds candidate test URLs:
   • baseUrl + "/chat/completions"
   • If baseUrl does not contain a version number, also try baseUrl + "/v1/chat/completions"
6. It sends POST requests to candidate URLs in order.
7. If any URL succeeds, it returns success.
8. If all URLs fail, it returns the last failure reason.

Notes:

• This is the only Provider management API that directly calls an external LLM API.
• The test sends a real HTTP request.
• HTTP errors record status code and response body, while other exceptions record exception type and message.

Default Provider Management

GET /api/llm-provider/default-provider Get Default Providers

Returns:

• Result<DefaultProviderDTO>

Flow:

1. Service acquires the read lock.
2. In DB mode, it queries globalSettingRepository.findById(1L).
3. It returns the default Chat Provider ID and default Embedding Provider ID.
4. In Legacy mode, it builds the response from properties.defaultProvider and properties.defaultEmbeddingProvider.

Response structure:

{
  "defaultProvider": "dashscope",
  "defaultEmbeddingProvider": "dashscope"
}

PUT /api/llm-provider/default-provider Set Default Chat Provider

Returns:

• Result<Void>

Flow:

1. Service starts a transaction and acquires the write lock.
2. It reads request.defaultProvider().
3. If the default Provider is empty, it throws BAD_REQUEST.
4. It queries the target Provider and confirms it exists.
5. In DB mode, it updates GlobalSettingEntity.defaultChatProviderId.
6. It saves global settings.
7. It calls registry.reload().

Legacy mode handling:

• Validates that the Provider exists.
• Updates properties.setDefaultProvider(providerId).
• Calls writeDefaultProviderToYaml(providerId) to write configuration back.
• Removes the old module-defaults configuration.
• Calls registry.reload().

PUT /api/llm-provider/default-embedding-provider Set Default Embedding Provider

Returns:

• Result<Void>

Flow:

1. Service starts a transaction and acquires the write lock.
2. It reads request.defaultEmbeddingProvider().
3. If the default Embedding Provider is empty, it throws BAD_REQUEST.
4. It queries the target Provider and confirms it exists.
5. It validates that the Provider supports Embedding:
   • supportsEmbedding must be true.
   • embeddingModel must exist.
   • validateEmbeddingConfig(...) must pass.
6. In DB mode, it updates GlobalSettingEntity.defaultEmbeddingProviderId.
7. It saves global settings.
8. It calls registry.reload().

Difference from default Chat Provider:

• Setting the default Embedding Provider includes additional Embedding capability validation.
• A Provider that does not support Embedding cannot be set as the default vector service.

ASR Configuration Management

GET /api/llm-provider/voice/asr Get ASR Configuration

Returns:

• Result<AsrConfigDTO>

Flow:

1. Service acquires the read lock.
2. It reads voiceProperties.getQwen().getAsr() from VoiceInterviewProperties.
3. It builds AsrConfigDTO:
   • url
   • model
   • language
   • format
   • sampleRate
   • maskedApiKey
   • enableTurnDetection
   • turnDetectionType
   • turnDetectionThreshold
   • turnDetectionSilenceDurationMs
   • VAD-related parameters
4. It returns the masked ASR configuration.

Notes:

• ASR configuration comes from VoiceInterviewProperties.
• The configuration prefix is app.voice-interview.
• This configuration does not use DB storage.

PUT /api/llm-provider/voice/asr Update ASR Configuration

Returns:

• Result<Void>

Flow:

1. Service acquires the write lock.
2. It reads runtime ASR and TTS configuration references.
3. It updates ASR fields selectively:
   • url
   • model
   • language
   • format
   • sampleRate
   • enableTurnDetection
   • turnDetectionType
   • turnDetectionThreshold
   • turnDetectionSilenceDurationMs
4. If API Key is updated, it synchronizes ASR and TTS:

asr.setApiKey(apiKey);
tts.setApiKey(apiKey);
updateEnvValue("AI_BAILIAN_API_KEY", apiKey);

5. It calls writeAsrConfigToYaml(asr) to write back to YAML.
6. It calls asrService.reload(voiceProperties) to reload ASR.
7. If API Key is updated, it also calls ttsService.reload(voiceProperties).

Notes:

• This method does not have @Transactional.
• ASR and TTS share the Bailian API Key.
• Updating the ASR API Key also affects TTS.

TTS Configuration Management

GET /api/llm-provider/voice/tts Get TTS Configuration

Returns:

• Result<TtsConfigDTO>

Flow:

1. Service acquires the read lock.
2. It reads voiceProperties.getQwen().getTts() from VoiceInterviewProperties.
3. It builds TtsConfigDTO:
   • model
   • maskedApiKey
   • voice
   • format
   • sampleRate
   • mode
   • languageType
   • speechRate
   • volume
4. It returns the masked TTS configuration.

PUT /api/llm-provider/voice/tts Update TTS Configuration

Returns:

• Result<Void>

Flow:

1. Service acquires the write lock.
2. It reads runtime ASR and TTS configuration references.
3. It updates TTS fields selectively:
   • model
   • voice
   • format
   • sampleRate
   • mode
   • languageType
   • speechRate
   • volume
4. If API Key is updated, it synchronizes TTS and ASR:

tts.setApiKey(apiKey);
asr.setApiKey(apiKey);
updateEnvValue("AI_BAILIAN_API_KEY", apiKey);

5. It calls writeTtsConfigToYaml(tts) to write back to YAML.
6. It calls ttsService.reload(voiceProperties) to reload TTS.
7. If API Key is updated, it also calls asrService.reload(voiceProperties).

Notes:

• TTS update logic is symmetrical with ASR.
• ASR/TTS API Keys are always updated together.

ASR Connectivity Test

POST /api/llm-provider/voice/asr/test Test ASR Connection

Returns:

• Result<ProviderTestResult>

Flow:

1. Service acquires the read lock.
2. It reads ASR configuration from voiceProperties.getQwen().getAsr().
3. It parses the WebSocket URL:
   • wss defaults to port 443.
   • ws defaults to port 80.
4. It runs a TCP Socket connection test:

socket.connect(address, 5000);
socket.close();

5. On success, it returns:

ProviderTestResult(success=true, "ASR WebSocket connection succeeded: host")

6. On failure, it returns the failure reason.

Differences from Provider connectivity testing:

• ASR testing only checks TCP Socket connectivity.
• It does not send a WebSocket handshake.
• It does not call the real ASR recognition API.
• Provider testing sends a real HTTP request to the LLM service.

Cache and Runtime Behavior

After Provider configuration changes, registry.reload() is called. This method clears internal caches:

clientCache.clear();
embeddingModelCache.clear();

Therefore, configuration changes do not immediately create new clients. Instead, clients are rebuilt on demand the next time business code uses the Provider. This avoids forcing update APIs to bear the initialization cost of model clients and ensures old configuration does not remain in cache for too long.

It is important to note that reload only clears caches; it does not synchronize configuration sources. If DB mode is enabled, the system prioritizes database configuration instead of re-importing from YAML or .env.

Current Issues and Optimization Directions

The module already supports DB mode and Legacy mode, but the configuration synchronization boundary still needs to be clarified:

• After DB mode is enabled, changes to YAML and .env are not automatically written back to the database.
• Restarting the project only reloads runtime configuration and cannot resolve inconsistencies between DB configuration and file configuration.
• Manual reload only clears runtime caches and does not re-import configuration sources.
• ASR/TTS configuration still comes from VoiceInterviewProperties, which is not the same storage as Provider DB configuration.
• ASR/TTS update methods have no transaction, so writing YAML, writing .env, and reloading services may partially succeed.

Future improvements:

• Add a configuration import API for DB mode to sync Providers from YAML and .env into the database.
• Add a one-time startup migration strategy and clearly define whether DB or configuration files have priority.
• Add a version number or update time to Provider configuration to help diagnose whether caches have refreshed.
• Move ASR/TTS configuration into unified configuration storage to reduce inconsistencies caused by multiple sources.
• Add failure compensation or clearer error messages for YAML writes, .env writes, and service reloads.

Summary

The llm-provider module is the unified configuration entry for large model capabilities. It manages not only Chat Providers, but also Embedding Providers, default models, runtime caches, and voice ASR/TTS configuration. Its key value is decoupling model configuration from business calls, allowing upper-layer features such as knowledgebase, RAG chat, and voice interview to obtain model capabilities through the unified Provider registry. The next focus is to further clarify DB and file-configuration synchronization so configuration sources are clearer and runtime state is more controllable.