CE-Prompt: enhance prompt expression stability by multiple understanding

Generative pre-trained models

Generative pre-trained (GPT) models (Radford et al., 2018) are pre-trained on large-scale datasets to learn language patterns and semantic relationships (Ouyang et al., 2022), enabling them to handle complex linguistic structures. Fine-tuning adapts these models to specific tasks, preserving their general language capabilities while optimizing performance on downstream applications.

The development of GPT builds on earlier advancements like Embeddings from Language Models (ELMo) (Ilić et al., 2018), which addressed polysemy by dynamically modeling word meanings in context. However, ELMo’s reliance on recurrent neural networks (RNNs) (Zaremba, Sutskever & Vinyals, 2014) limited its scalability due to sequential computation. The introduction of the attention mechanism revolutionized NLP by enabling parallel processing, leading to the GPT model. GPT leverages the Transformer’s Masked Self-Attention (Li & Liang, 2021) and adopts an auto-regressive approach, where each token prediction depends on previously generated tokens, enhancing both efficiency and language understanding.

Recent advancements have produced powerful auto-regressive models like Large Language Model Meta AI (LLaMA) (Touvron et al., 2023), GPT-3 (Brown et al., 2020), T5 (Raffel et al., 2020), General Language Model (GLM) (Du et al., 2021), and Qwen (Bai et al., 2023). These models excel in tasks such as text generation, summarization, translation, and question answering (McCann et al., 2018), thanks to their extensive pre-training and scalable architectures. Notably, GPT-3 introduced few-shot and zero-shot learning capabilities, reducing the need for task-specific fine-tuning. Additionally, models like T5 unified diverse NLP tasks into a text-to-text framework, further simplifying their application.

The success of these models stems from their ability to leverage massive datasets, advanced architectures, and computational resources. However, challenges remain, such as high computational costs and the need for more efficient training methods, which are active areas of research.

Prompt-Engineer

In the evolution of prompt engineering (Sahoo et al., 2024), researchers initially proposed Zero-shot Prompting and Few-shot Prompting to address new tasks without extensive training, enabling inference with minimal or no examples. However, as tasks grew more complex, models still struggled with logical reasoning and multi-step thinking. This led to the development of Chain-of-Thought (CoT) (Wei et al., 2022), followed by extensions like Automatic Chain-of-Thought (Auto-CoT) (Zhang et al., 2022) and Self-Consistency to improve reasoning accuracy. Further innovations, such as Logical CoT (LogiCoT) (Liu et al., 2023), Chain-of-Symbol (CoS) (Hu et al., 2023), Tree-of-Thoughts (ToT) (Yao et al., 2024), Graph-of-Thought (GoT) (Besta et al., 2024), Thread of Thought (ThoT) (Zhou et al., 2023), and Chain of Table Prompting, provided structured solutions for complex logical tasks.

To reduce hallucinations, researchers integrated external retrieval and verification into methods like Retrieval-Augmented Generation (RAG) (Zhao et al., 2024), ReAct (Yao et al., 2022), Chain-of-Verification (CoVe) (Dhuliawala et al., 2023), Chain-of-Note (CoN) (Yu et al., 2023), and Chain-of-Knowledge (CoK) (Li et al., 2023), leveraging external knowledge or self-verification to minimize errors. For user-model interaction, Active-Prompt allows dynamic guidance, while Automatic Prompt Engineer (APE) automates prompt construction, reducing manual effort.

In knowledge-driven reasoning, Automatic Reasoning and Tool-use (ART) (Paranjape et al., 2023) and Contrastic Chain-of-Thought (CCoT) (Chen et al., 2022) enhance reasoning by integrating knowledge bases or toolchains. Efforts to improve textual consistency and readability ensure logical coherence, while Emotion Prompting incorporates emotional cues for better user interaction.

For code generation, techniques like Scratchpad Prompting, Program of Thoughts (PoT) (Wang et al., 2018), Structured Chain-of-Thought (SCoT) (Li et al., 2025), and Chain-of-Code (CoC) (Li et al., 2023) embed programming logic, expanding models’ applicability in software development. Research on optimization by prompting improves inference efficiency, while Rephrase and Respond (RaR) (Deng et al., 2023) and Take a Step Back Prompting enable iterative self-correction, fostering higher levels of intelligence and self-reflection.

Parameter-efficient fine-tuning

In the fine-tuning process of large-scale pre-trained models, efficiently adapting to downstream tasks has become a critical issue. To address the computational overhead and storage costs, a variety of Parameter-Efficient Fine-Tuning methods have been proposed in recent years. These methods aim to reduce the number of parameters that need to be adjusted while maintaining the model’s task adaptation capabilities.

One strategy for efficient fine-tuning is modifying the model structure. For instance, Adapter-Tuning inserts adapter layers between the Transformer blocks, enabling task-specific adaptations without altering the original model parameters. This approach offers a high level of parameter efficiency. Another common method involves fine-tuning only a subset of the model’s parameters. LoRA achieves this by injecting low-rank matrices, reducing the number of parameters that need to be updated. This technique leverages matrix factorization to efficiently update the model, thereby reducing computational costs. Additionally, methods like Prefix-Tuning and Prompt-Tuning adapt the model by adding task-specific virtual tokens or soft prompt embeddings to the input. These approaches update only the prefix of the input, without altering the internal parameters of the model, thus providing a lightweight adaptation mechanism.

Building upon these foundational methods, several extensions have been proposed, such as DoRA (Liu et al., 2024), Multitask Prompt-Tuning (Wang et al., 2023), and P-Tuning v2 (Liu et al., 2021). While these methods have shown certain advantages in specific scenarios, they also come with limitations, such as strong dependency on specific tasks or unstable performance when handling long-range dependencies.

Overall, the research on PEFT continues to evolve, with new methods offering diverse options for efficient fine-tuning of large-scale pre-trained models. The central challenge remains how to reduce the number of parameters adjusted while improving the model’s performance.

Datasets and tasks

In this study, we categorize the datasets into two types according to their experimental purposes: the primary dataset, which is used for model comparison, sensitivity analysis, and clustering visualization; and the auxiliary datasets, which are used to analyze the training dynamics under different prompt initialization strategies.

Model

We select LlaMA3-8B-Instruct and Qwen2.5-7B-Instruct as the base models.

LlaMA3-8B-Instruct: LlaMA3-8B-Instruct is a third-generation LlaMA model with around 8 billion parameters. By using a more diverse pre-training dataset and instruction tuning, it improves instruction following, contextual reasoning, and handling of complex queries. It also enhances long-text comprehension and reduces contradictory outputs.

Qwen2.5-7B-Instruct: Qwen2.5-7B-Instruct is a multilingual language model with around 7 billion parameters. It focuses on Chinese, English, and other multilingual tasks, leveraging broader domain data and diverse training approaches. Optimized for cross-lingual tasks, it provides accurate, context-consistent outputs and offers faster inference and lower deployment costs.

Baseline

In this study, we established a baseline by performing Prompt-Tuning on two advanced pre-trained language models, LlaMA3-8B-Instruct and Qwen2.5-7B-Instruct. These models, known for their outstanding language generation capabilities and task adaptability, provided a reliable foundation for comparison with our proposed methods.

For the baseline evaluation on medical domain data, we utilized BLEU-4 (Papineni et al., 2002) and ROUGE-L (Lin, 2004) as the evaluation metrics.

BLEU-4 is a widely used metric for evaluating machine-generated text by comparing it with human-written reference texts. It calculates the precision of n-grams (Majumder, Mitra & Chaudhuri, 2002) while incorporating a brevity penalty (BP) to prevent bias toward shorter sentences. The BLEU score is computed using the following formula:

(9) $$BLEU=BP\cdot \exp \left(\sum _{n=1}^4{w}_n\log p_n\right)$$where $p_n$ represents the precision of n-grams, $w_n$ is the weight assigned to each n-gram (typically, equal weights for $n=1$ to $4$), the BLEU score is expressed by the formula:

(10) $$BP=\{\begin{array}{cc}1,\hfill & \mathrm{i}\mathrm{f}c>r\hfill \\ e^{(1-r/c)},\hfill & \mathrm{i}\mathrm{f}c\le r\hfill \end{array}$$where $c$ is the length of the generated text, and $r$ is the length of the reference text. A higher BLEU-4 score indicates better alignment with the reference text in terms of fluency and accuracy.

ROUGE-L measures the similarity between generated text and reference text based on the Longest Common Subsequence (LCS). Unlike BLEU, which is precision-based, ROUGE-L emphasizes recall, capturing the overall sentence structure similarity. The formula for ROUGE-L is given by:

(11) $$ROUGE\text{-}L=\frac{LCS(X,Y)}{max(|X|,|Y|)}$$where $LCS(X,Y)$ is the longest common subsequence between the generated text X and the reference text Y, |X| and |Y | denote the lengths of the generated and reference texts, respectively.

ROUGE-L provides insights into the coherence and relevance of generated text, particularly useful for evaluating dialogue systems and summarization tasks. A higher ROUGE-L score indicates that the generated text better preserves the structure and meaning of the reference response.

These evaluation metrics help assess the fluency, coherence, and relevance of generated medical dialogues, ensuring an objective comparison with reference responses.

Implementation details

To comprehensively evaluate the proposed CE-Prompt method, we design three core experiments focusing on overall performance, parameter sensitivity, and representation ability.

Comparative experiments

To comprehensively evaluate the performance of the proposed CE-Prompt method on generative tasks, we conduct systematic comparisons with several mainstream parameter-efficient fine-tuning approaches, including P-Tuning, Prefix-Tuning, Prompt-Tuning, and LoRA. All methods are implemented on two representative large language models, LLaMA and Qwen, and evaluated using BLEU-4 and ROUGE-L metrics to measure the accuracy and semantic coverage of generated outputs. The detailed results are shown in Table 1.

As shown in the figure, CE-Prompt achieves the best performance across both model backbones. On the LLaMA model, it reaches a BLEU-4 score of 91.09% and a ROUGE-L score of 93.64%, outperforming all baseline methods. Similarly, on the Qwen model, CE-Prompt yields a BLEU-4 score of 89.25% and a ROUGE-L score of 93.39%, again ranking highest among all approaches. These results suggest that the multi-embedding mechanism employed in CE-Prompt effectively enhances the model’s adaptability to task-specific features, leading to improvements in both generation quality and consistency.

It is worth noting that Prompt-Tuning requires fewer trainable parameters than LoRA and Prefix-Tuning. For instance, on the Qwen model, Prompt-Tuning achieves a BLEU-4 score of 88.35% and a ROUGE-L score of 92.70%, closely matching LoRA’s performance (88.96%/93.04%). A similar trend is observed on the LLaMA model. This indicates that for tasks like Text2Cypher, where the semantic structure is relatively well-defined and the domain boundaries are clear, Prompt-Tuning can sufficiently activate the language modeling capabilities already embedded within the pretrained model, achieving competitive results with minimal parameter updates.

Overall, while CE-Prompt demonstrates strong performance advantages due to its composite embedding strategy, these results also suggest that Prompt-Tuning remains a viable and efficient fine-tuning method in structured generation tasks, particularly when computational resources or parameter budgets are constrained.

Sensitivity analysis

To thoroughly evaluate the behavior of CE-Prompt under different configurations, we conducted extended sensitivity experiments focusing on two key factors: prompt length and the number of composite embeddings (M). Experiments were performed on both LLaMA and Qwen models across various configurations, and results were compared against standard Prompt-Tuning. The findings are summarized in Tables 2 and 3.

From the perspective of prompt length, both Prompt-Tuning and CE-Prompt show performance improvements as the length increases. However, the gains tend to diminish with longer prompts. For example, on LLaMA, the BLEU-4 score of Prompt-Tuning improves from 89.16% (Length = 16) to 90.48% (Length = 64), but further increases to Length = 256 only yield a minor improvement to 90.76%. A similar saturation trend is observed across CE-Prompt settings with different M values. These two phenomena can be seen more clearly in the line chart of Fig. 5. This indicates that while longer prompts can enhance expressiveness to some extent, the marginal benefit decreases beyond a certain length.

Regarding the number of composite embeddings, we observe that increasing M generally leads to better performance under the same prompt length. For instance, under Length = 32 on LLaMA, BLEU-4 increases from 90.61% at $M=2$ to 90.94% at $M=4$ and reaches 91.22% at $M=8$. However, this improvement also becomes less significant as M increases. These results suggest that although increasing M enhances representation capacity, the performance gain gradually stabilizes, and moderate values such as $M=4$ or $M=8$ offer a good balance between effectiveness and computational cost.

The two hyperparameters in the table determine the final number of trainable parameters for CE-Prompt. When Length * M is equal, the number of trainable parameters is the same, allowing us to make comparisons under these equal parameter settings. In this case, CE-Prompt consistently outperforms Prompt-Tuning, demonstrating that its performance advantage is not merely due to parameter scaling. Instead, the design of CE-Prompt—especially its composite embedding mechanism—plays a critical role in enhancing effectiveness. Moreover, while Prompt-Tuning relies on extending the prompt length to improve performance, this also increases the input sequence length, leading to higher computational and inference costs. In contrast, CE-Prompt achieves comparable or better performance under the same parameter budget using shorter prompt lengths, thereby reducing latency and improving efficiency.

Clustering analysis

To further understand the representational differences between CE-Prompt and traditional Prompt-Tuning, we conducted a clustering visualization experiment based on the learned prompt vectors. After training, we extracted the token-level prompt embeddings and applied principal component analysis (PCA) to reduce their dimensionality for visualization. The distributions of the token vectors under LLaMA and Qwen models are shown in Fig. 6.

In Fig. 6A, the token vectors from CE-Prompt are more widely distributed across the feature space, indicating higher dispersion and semantic diversity. This suggests that CE-Prompt enables each token to learn more independent and distinct semantic representations, enriching the overall expressiveness of the prompt. In contrast, the token vectors from Prompt-Tuning are densely clustered in the lower-left region, implying that the learned embeddings are concentrated and semantically homogeneous. Such limited variation may restrict the model’s ability to encode nuanced task-specific features.

In Fig. 6B, a similar trend is observed. Although the separation in representation space is relatively less pronounced compared to LLaMA, CE-Prompt still yields a broader distribution of token vectors than Prompt-Tuning. This highlights the generalizability of CE-Prompt across different model architectures, as it consistently enhances embedding diversity.

Trainable parameter analysis

In terms of trainable parameter size, although CE-Prompt introduces more parameters than Prompt-Tuning, the overall proportion remains negligible relative to the base model size. As shown in Table 1, CE-Prompt accounts for only 0.013% and 0.012% of total parameters on LLaMA and Qwen, respectively—slightly higher than Prompt-Tuning (0.002%) but significantly lower than LoRA, which consumes 0.259% and 0.265% on the same models. Notably, despite using fewer parameters, CE-Prompt consistently outperforms LoRA in both BLEU-4 and ROUGE-L scores, achieving the best overall generation quality across both backbones. This indicates that CE-Prompt achieves a superior trade-off between parameter scale and performance. Its ability to deliver strong results with only a minimal number of updated parameters makes it particularly attractive for efficient and scalable deployment in practice.

Time complexity analysis

In terms of computational cost, the decoder stage of Transformer models is primarily dominated by multi-head self-attention, with an overall time complexity of $O(n^2\cdot d)$, where $n$ is the sequence length and $d$ is the embedding dimension. Both Prompt-Tuning and CE-Prompt increase inference input length by prepending prompt embeddings, and thus their time complexity grows with $O(L+N)$, where L is the prompt length and N is the task input length. However, CE-Prompt is more efficient, as it achieves stronger performance with a shorter prompt.

Based on the sensitivity experiments and assuming an average input length of 40 tokens, Prompt-Tuning typically requires a prompt length of 256 to achieve optimal performance, while CE-Prompt achieves comparable or better results with a prompt length of 32 and $K=4$ composite embeddings. The resulting total sequence lengths are:

• Prompt-Tuning: $n=256+40=296$, with complexity $O({296}^2\cdot d)$

• CE-Prompt: $n=32+40=72$, with complexity $O({72}^2\cdot d).$

This implies that CE-Prompt requires only about $({72}^2/{296}^2)\approx 5.9\mathrm{\%}$ of the attention-related computation compared to Prompt-Tuning, demonstrating a significant advantage in inference efficiency.

Appendix A.1 Llama3-70B system content illustration. (Since the dataset is in Chinese, it needs to be localized into Chinese according to the following template.)

You are an expert in knowledge graphs. There is an existing medical-related knowledge graph. The entities in the graph are defined as follows:

Entities

• disease: Disease, storing basic information about various diseases.

• department: Department, the medical department associated with a disease.

• symptom: Symptoms of a disease.

• check: Medical examinations related to a disease.

• drug: Medications used for treating a disease.

• producer: Drug product available for sale.

• food: Food, including recommended and restricted food items.

Relationships Between Entities

Each relationship can be bidirectional, where $v$ represents an entity and $e$ represents a relationship:

• Disease-Department Relationship: $(v:\mathrm{D}\mathrm{i}\mathrm{s}\mathrm{e}\mathrm{a}\mathrm{s}\mathrm{e})-[e:\mathrm{b}\mathrm{e}\mathrm{l}\mathrm{o}\mathrm{n}\mathrm{g}\mathrm{s}\mathrm{\_}\mathrm{t}\mathrm{o}]\to (v:\mathrm{D}\mathrm{e}\mathrm{p}\mathrm{a}\mathrm{r}\mathrm{t}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{t})$

• Disease-Symptom Relationship: $(v:\mathrm{D}\mathrm{i}\mathrm{s}\mathrm{e}\mathrm{a}\mathrm{s}\mathrm{e})-[e:\mathrm{h}\mathrm{a}\mathrm{s}\mathrm{\_}\mathrm{s}\mathrm{y}\mathrm{m}\mathrm{p}\mathrm{t}\mathrm{o}\mathrm{m}]\to (v:\mathrm{S}\mathrm{y}\mathrm{m}\mathrm{p}\mathrm{t}\mathrm{o}\mathrm{m})$

• Disease-Drug Relationship: $(v:\mathrm{D}\mathrm{i}\mathrm{s}\mathrm{e}\mathrm{a}\mathrm{s}\mathrm{e})-[e:\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{m}\mathrm{o}\mathrm{n}\mathrm{\_}\mathrm{d}\mathrm{r}\mathrm{u}\mathrm{g}]\to (v:\mathrm{D}\mathrm{r}\mathrm{u}\mathrm{g})$; $(v:\mathrm{D}\mathrm{i}\mathrm{s}\mathrm{e}\mathrm{a}\mathrm{s}\mathrm{e})-[e:\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{m}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{\_}\mathrm{d}\mathrm{r}\mathrm{u}\mathrm{g}]\to (v:\mathrm{D}\mathrm{r}\mathrm{u}\mathrm{g})$

• Disease-Check Relationship: $(v:\mathrm{D}\mathrm{i}\mathrm{s}\mathrm{e}\mathrm{a}\mathrm{s}\mathrm{e})-[e:\mathrm{n}\mathrm{e}\mathrm{e}\mathrm{d}\mathrm{\_}\mathrm{c}\mathrm{h}\mathrm{e}\mathrm{c}\mathrm{k}]\to (v:\mathrm{c}\mathrm{h}\mathrm{e}\mathrm{c}\mathrm{k})$

• Producer-Drug Relationship: $(v:\mathrm{P}\mathrm{r}\mathrm{o}\mathrm{d}\mathrm{u}\mathrm{c}\mathrm{e}\mathrm{r})-[e:\mathrm{d}\mathrm{r}\mathrm{u}\mathrm{g}\mathrm{s}\mathrm{\_}\mathrm{o}\mathrm{f}]\to (v:\mathrm{D}\mathrm{r}\mathrm{u}\mathrm{g})$

• Disease-Food Relationship: $(v:\mathrm{D}\mathrm{i}\mathrm{s}\mathrm{e}\mathrm{a}\mathrm{s}\mathrm{e})-[e:\mathrm{d}\mathrm{o}\mathrm{\_}\mathrm{e}\mathrm{a}\mathrm{t}]\to (v:\mathrm{F}\mathrm{o}\mathrm{o}\mathrm{d})$; $(v:\mathrm{D}\mathrm{i}\mathrm{s}\mathrm{e}\mathrm{a}\mathrm{s}\mathrm{e})-[e:\mathrm{n}\mathrm{o}\mathrm{\_}\mathrm{e}\mathrm{a}\mathrm{t}]\to (v:\mathrm{F}\mathrm{o}\mathrm{o}\mathrm{d})$; $(v:\mathrm{D}\mathrm{i}\mathrm{s}\mathrm{e}\mathrm{a}\mathrm{s}\mathrm{e})-[e:\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{m}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{\_}\mathrm{e}\mathrm{a}\mathrm{t}]\to (v:\mathrm{F}\mathrm{o}\mathrm{o}\mathrm{d})$

• Disease-Complication Relationship: $(v:\mathrm{D}\mathrm{i}\mathrm{s}\mathrm{e}\mathrm{a}\mathrm{s}\mathrm{e})-[e:\mathrm{a}\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{p}\mathrm{o}\mathrm{n}\mathrm{y}\mathrm{\_}\mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}]\to (v:\mathrm{D}\mathrm{i}\mathrm{s}\mathrm{e}\mathrm{a}\mathrm{s}\mathrm{e})$

Main Attributes of Entities

• disease: {name: disease name, desc: disease description, cause: causes of the disease, prevent: prevention measures, cure_lasttime: treatment duration, cure_way: treatment method, cured_prob: cure probability, easy_get: susceptible population}

• department: {name: department name}

• symptom: {name: symptom name}

• check: {name: check item}

• drug: {name: drug name}

• producer: {name: drug name}

• food: {name: food name}

Query Generation

Based on the above background and the user’s query, generate an OpenCypher query statement to accurately retrieve relevant knowledge from the graph.

Note:

• Use only the provided entities, relationships, and attributes. Do not include any additional information.

• Relationships between entities should strictly follow the provided relationships.

(Output only the OpenCypher query, without explanation.)

Appendix B.1 Knowledge graph entity-related categories. (Since the dataset is in Chinese, it needs to be localized into Chinese according to the following template.)

Appendix B.2 Knowledge graph relationship-related categories. (Since the dataset is in Chinese, it needs to be localized into Chinese according to the following template.)

Appendix C.1 The following are the prompt texts used for text initialization in the comparison of loss trends across four tasks:

• CoLA (Corpus of Linguistic Acceptability):

  Please judge the correctness or error of the sentence syntax according to the input sentence content output right or error.

• RTE (Recognizing Textual Entailment):

  Please output an entailment or not_entailment according to the contents of sentence1 and sentence2 determine whether the two sentences have an entailment relationship.

• MMLU (Massive Multitask Language Understanding):

  Please select the correct answer from the choices based on the content of the question.

• IMDB (Movie Review Sentiment Analysis):

  Please judge whether the sentence is positive or negative based on the content of the user review.