DeepSeek-V4 Preview Released — 1.6 Trillion Parameter MoE with 1 Million Token Context
DeepSeek-AI has released a preview of the DeepSeek-V4 series. It consists of two Mixture-of-Experts (MoE) models: DeepSeek-V4-Pro with 1.6 trillion parameters (49B activated at inference time) and DeepSeek-V4-Flash with 284 billion parameters (13B activated). Both models support a context length of 1 million tokens.
On the architecture side, three major improvements are incorporated. First, a hybrid attention architecture combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) improves long-context efficiency. Second, Manifold-Constrained Hyper-Connections (mHC) enhance conventional residual connections. Third, the adoption of the Muon optimizer improves convergence speed and training stability.
The models were pre-trained on more than 32 trillion tokens of diverse, high-quality data. DeepSeek-V4-Pro-Max, the maximum inference mode, set a new state-of-the-art for open models at the time. In long-context scenarios in particular, it reduces per-token inference FLOPs to 27% and KV cache to 10% compared to DeepSeek-V3.2, making the everyday use of 1 million tokens practical.
Model checkpoints are available on Hugging Face.
White House and Anthropic in Talks on Crafting AI Safety Rules
According to Politico, talks between the White House and Anthropic are underway to develop AI safety rules. For the specific contents of the rules and their enforcement timeline, you'll need to refer to the original article — but the fact that dialogue between AI companies and government agencies is shifting into the stage of practical rule-making from a national security perspective is worth noting.
Comprehensive Benchmark Analysis of Diffusion Language Models (DLMs)
A paper titled "Diffusion Language Models: An Experimental Analysis" reports the results of systematically evaluating eight recent DLMs across eight benchmarks (reasoning, coding, translation, knowledge, and structured problem solving).
Unlike conventional autoregressive LLMs, DLMs generate entire sentences in parallel through iterative denoising. The study demonstrates that inference-time design choices (the number of denoising steps, context length, block size, and parallel unmasking strategies) significantly affect model behavior. It also provides a detailed analysis of DLM architectures and the performance/compute-efficiency trade-offs at different inference budgets, offering practical insights for future DLM deployment.
"Emergent Alignment" — LLMs Detect and Correct Their Own Ethical Issues
"Emergent Alignment" proposes a mechanism that incorporates a "conscience step" into LLMs, letting them review their own outputs from an ethical perspective.
Specifically, it adds a process in which the model self-checks its own reasoning and outputs, and extends the training loss with an alignment component based on Direct Preference Optimization (DPO). This steers the model away from unethical outputs. Interestingly, it requires no external judge model, relying instead on a frozen copy of the model itself.
Against the conventional "Emergent Misalignment" scenario (where models fine-tuned on code hacking exhibit a variety of unethical behaviors), the authors demonstrated that a single high-level introspective question can guide models toward ethical behavior even under the same code-hacking conditions.
Flourish Labs Raises $500 Million to Reinvent AI with Neuroscience
According to a Wired article (PDF), Flourish Labs has raised $500 million for its effort to fundamentally rethink AI based on neuroscience. As a push toward new AI architectures inspired by how the human brain works, this approach — distinct from existing transformer-based methods — is drawing attention.
Human-like Driving Agents from Minimal Human Data
A paper titled "Human-like autonomy emerges from self-play and a pinch of human data" shows that adding just 30 minutes of human driving demonstrations to self-play reinforcement learning is enough to learn driving policies that can coordinate with humans.
To address the problem that pure self-play learns driving conventions that are effective but alien to humans, human data is used for regularization purposes. Compared with an equivalent imitation learning approach, it needs only 1/2500th of the human data, and training completes in 15 hours on a single consumer GPU. Source code and videos have also been released.
---