"We are reaching the limits of our patience here," Panetta said during a joint press conference the Afghan defense minister.
A senior U.S. official acknowledged Thursday that the recent increase in drone strikes on insurgents in Pakistan is due in part to frustration with Islamabad. Despite pressure from U.S. commanders, Pakistan remains reluctant to go after insurgents, particularly the Haqqani network, which was likely responsible for recent deadly attacks on U.S. forces.
Panetta added that Pakistan must take action and that there is "an increasing concern that this safe haven exists," Panetta said. "It is very important that Pakistan take steps to deal with this threat."
Lawmakers from both parties have been urging the State Department to designate the Haqqani network a foreign terrorist organization.
Panetta's stop in Kabul caps two days of blunt commentary on Pakistan.
Panetta arrived in Afghanistan on Thursday to take stock of progress in the war and discuss plans for the troop drawdown, even as violence spiked in the south.
He acknowledged the increase in attacks in Afghanistan and that the insurgents appear to be much more organized. But he insisted that the overall level of violence was down, and that commanders had expected the uptick.
Panetta said he wants to get an assessment of the situation from the top U.S. commander, Marine Gen. John Allen, and see how confident he is about NATO's ability to confront the threats both from the Taliban and the Haqqani network.
"I think it's important to make sure we are aware of the kind of attacks they're going to engage in ... as we go through the rest of the summer," Panetta told reporters traveling with him during a stop in New Delhi, India, on Wednesday.
Panetta is arriving just a day after three suicide attackers blew themselves up in a marketplace in southern Afghanistan, killing 22 people and wounding at least 50 others. In the east, meanwhile, Afghan officials and residents said a predawn NATO airstrike targeting militants killed civilians celebrating a wedding, including women and children. A NATO forces spokesman said the coalition had no reports of civilians being killed in a raid.
Allen has to withdraw 23,000 American troops by the end of September, leaving about 68,000 U.S. military personnel in the country. Officials have said the bulk of the 23,000 probably will not come out until shortly before the deadline.
As those troops leave, Allen has said that Afghan forces will be used to fill in the gaps in the eastern and southwestern parts of the country. They will be buttressed by U.S. advisory teams that will work with the Afghan units.
Once the 23,000 U.S. troops depart, Allen is expected to review how the fighting season is going and then will begin to put together an analysis for President Barack Obama on how troop withdrawals will proceed next year.
Panetta also was scheduled to meet with U.S. Ambassador Ryan Crocker.
The defense secretary also joked with troops at the Kabul airport about the U.S. strike that killed an Al Qaeda leader Monday, saying, "the worst job you can get these days is to be a deputy leader in Al Qaeda."